Oct 31 21:13:17.222116 ip-10-0-131-21 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 21:13:17.222131 ip-10-0-131-21 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 21:13:17.222142 ip-10-0-131-21 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 21:13:17.222559 ip-10-0-131-21 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 21:13:27.252244 ip-10-0-131-21 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Oct 31 21:13:27.252257 ip-10-0-131-21 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 96a4953a69d640a08e2608435db256a4 -- Oct 31 21:15:10.298747 ip-10-0-131-21 systemd[1]: Starting Kubernetes Kubelet... Oct 31 21:15:10.711917 ip-10-0-131-21 kubenswrapper[2639]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:10.711917 ip-10-0-131-21 kubenswrapper[2639]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 21:15:10.711917 ip-10-0-131-21 kubenswrapper[2639]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:10.711917 ip-10-0-131-21 kubenswrapper[2639]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:10.719487 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719333 2639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:10.719487 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719482 2639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:10.719487 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719488 2639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:10.719487 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719491 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:10.719487 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719495 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:10.719487 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719498 2639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:10.719487 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719501 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719504 2639 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719507 2639 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719510 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719513 2639 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719516 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719519 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719521 2639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719525 2639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719529 2639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719534 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719538 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719541 2639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719544 2639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719547 2639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719550 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719553 2639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719556 2639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719558 2639 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:10.719757 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719561 2639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719564 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719566 2639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719569 2639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719572 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719574 2639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719583 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719586 2639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719589 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719591 2639 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719594 2639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719596 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719599 2639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719602 2639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719606 2639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719611 2639 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719614 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719617 2639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719620 2639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:10.720213 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719623 2639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719626 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719629 2639 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719632 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719634 2639 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719637 2639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719639 2639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719642 2639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719644 2639 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719647 2639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719650 2639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719652 2639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719655 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719657 2639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719660 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719663 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719666 2639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719668 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719671 2639 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719674 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:10.720679 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719677 2639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719679 2639 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719682 2639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719685 2639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719687 2639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719690 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719692 2639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719695 2639 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719698 2639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719701 2639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719703 2639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719706 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719709 2639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719711 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719714 2639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719716 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719719 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719721 2639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719724 2639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719727 2639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:10.721160 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719730 2639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719732 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719735 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719737 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719740 2639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719742 2639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719745 2639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719748 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719750 2639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719753 2639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719755 2639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719759 2639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719762 2639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719765 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719769 2639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719772 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719775 2639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719777 2639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719780 2639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:10.721644 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719783 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719786 2639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.719788 2639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720675 2639 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720684 2639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720688 2639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720691 2639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720694 2639 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720697 2639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720700 2639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720702 2639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720705 2639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720708 2639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720711 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720714 2639 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720718 2639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720721 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720723 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720726 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720729 2639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:10.722130 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720732 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720735 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720737 2639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720740 2639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720743 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720745 2639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720748 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720751 2639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720753 2639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720755 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720758 2639 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720761 2639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720763 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720768 2639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720770 2639 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720773 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720775 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720778 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720781 2639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:10.722645 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720783 2639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720786 2639 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720788 2639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720791 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720794 2639 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720796 2639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720799 2639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720802 2639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720805 2639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720808 2639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720811 2639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720813 2639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720816 2639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720819 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720822 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720824 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720827 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720829 2639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720833 2639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:10.723113 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720837 2639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720841 2639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720843 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720846 2639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720849 2639 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720852 2639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720855 2639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720858 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720861 2639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720864 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720866 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720869 2639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720873 2639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720876 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720878 2639 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720881 2639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720884 2639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720886 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720889 2639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:10.723575 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720891 2639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720894 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720897 2639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720900 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720903 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720905 2639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720908 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720911 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720914 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720916 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720919 2639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720922 2639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720924 2639 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720927 2639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720929 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720932 2639 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720934 2639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720937 2639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720940 2639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:10.724034 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720942 2639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720945 2639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720948 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720950 2639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720953 2639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720955 2639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720958 2639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720960 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720963 2639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720965 2639 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720969 2639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720973 2639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.720977 2639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721554 2639 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721560 2639 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721564 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721568 2639 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721571 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721574 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721576 2639 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:10.724507 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721579 2639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721582 2639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721584 2639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721587 2639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721589 2639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721592 2639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721594 2639 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721597 2639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721600 2639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721603 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721605 2639 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721607 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721610 2639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721613 2639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721616 2639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721618 2639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721621 2639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721624 2639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721626 2639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:10.724993 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721629 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721631 2639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721634 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721636 2639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721639 2639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721641 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721645 2639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721649 2639 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721652 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721656 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721659 2639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721662 2639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721665 2639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721667 2639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721670 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721672 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721675 2639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721677 2639 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721680 2639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:10.725454 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721682 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721685 2639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721688 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721690 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721693 2639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721695 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721697 2639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721702 2639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721707 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721709 2639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721712 2639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721715 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721717 2639 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721720 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721722 2639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721725 2639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721727 2639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721730 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721732 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:10.725909 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721735 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721737 2639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721740 2639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721743 2639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721746 2639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721748 2639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721751 2639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721753 2639 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721756 2639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721758 2639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721761 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721763 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721766 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721768 2639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721771 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721773 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721776 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721778 2639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721781 2639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:10.726418 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721783 2639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721788 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721791 2639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721794 2639 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721796 2639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721799 2639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721802 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721804 2639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721807 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721809 2639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721812 2639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721815 2639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721817 2639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721820 2639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721822 2639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721825 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721828 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721830 2639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721834 2639 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721837 2639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:10.726901 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721840 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721842 2639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.721845 2639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723748 2639 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723760 2639 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723771 2639 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723776 2639 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723781 2639 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723785 2639 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723791 2639 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723795 2639 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723799 2639 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723802 2639 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723807 2639 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723810 2639 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723813 2639 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723816 2639 flags.go:64] FLAG: --cgroup-root="" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723819 2639 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723823 2639 flags.go:64] FLAG: --client-ca-file="" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723828 2639 flags.go:64] FLAG: --cloud-provider="external" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723832 2639 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723837 2639 flags.go:64] FLAG: --cluster-domain="" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723840 2639 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 21:15:10.727366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723844 2639 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723847 2639 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723851 2639 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723855 2639 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723859 2639 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723862 2639 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723866 2639 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723869 2639 flags.go:64] FLAG: --contention-profiling="false" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723873 2639 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723876 2639 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723880 2639 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723883 2639 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723887 2639 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723891 2639 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723894 2639 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723897 2639 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723900 2639 flags.go:64] FLAG: --enable-server="true" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723903 2639 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723908 2639 flags.go:64] FLAG: --event-burst="100" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723911 2639 flags.go:64] FLAG: --event-qps="50" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723914 2639 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723918 2639 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723921 2639 flags.go:64] FLAG: --eviction-hard="" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723927 2639 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 21:15:10.728006 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723930 2639 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723933 2639 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723936 2639 flags.go:64] FLAG: --eviction-soft="" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723940 2639 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723945 2639 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723948 2639 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723951 2639 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723954 2639 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723957 2639 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723960 2639 flags.go:64] FLAG: --feature-gates="" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723964 2639 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723968 2639 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723971 2639 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723975 2639 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723978 2639 flags.go:64] FLAG: --healthz-port="10248" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723981 2639 flags.go:64] FLAG: --help="false" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723984 2639 flags.go:64] FLAG: --hostname-override="ip-10-0-131-21.ec2.internal" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723988 2639 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723991 2639 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723995 2639 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.723998 2639 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724001 2639 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724005 2639 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 21:15:10.728644 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724007 2639 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724010 2639 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724013 2639 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724016 2639 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724019 2639 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724022 2639 flags.go:64] FLAG: --kube-reserved="" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724025 2639 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724028 2639 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724032 2639 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724035 2639 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724038 2639 flags.go:64] FLAG: --lock-file="" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724041 2639 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724044 2639 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724048 2639 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724055 2639 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724058 2639 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724061 2639 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724064 2639 flags.go:64] FLAG: --logging-format="text" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724067 2639 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724071 2639 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724073 2639 flags.go:64] FLAG: --manifest-url="" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724076 2639 flags.go:64] FLAG: --manifest-url-header="" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724081 2639 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724084 2639 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724089 2639 flags.go:64] FLAG: --max-pods="110" Oct 31 21:15:10.729208 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724092 2639 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724095 2639 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724098 2639 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724102 2639 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724105 2639 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724108 2639 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724111 2639 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724120 2639 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724123 2639 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724126 2639 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724129 2639 flags.go:64] FLAG: --pod-cidr="" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724132 2639 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724135 2639 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724138 2639 flags.go:64] FLAG: --pods-per-core="0" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724141 2639 flags.go:64] FLAG: --port="10250" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724144 2639 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724147 2639 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e5be499730184ae" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724151 2639 flags.go:64] FLAG: --qos-reserved="" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724154 2639 flags.go:64] FLAG: --read-only-port="10255" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724156 2639 flags.go:64] FLAG: --register-node="true" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724160 2639 flags.go:64] FLAG: --register-with-taints="" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724164 2639 flags.go:64] FLAG: --registry-burst="10" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724167 2639 flags.go:64] FLAG: --registry-qps="5" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724170 2639 flags.go:64] FLAG: --reserved-cpus="" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724173 2639 flags.go:64] FLAG: --reserved-memory="" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724177 2639 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 21:15:10.729841 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724181 2639 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724184 2639 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724187 2639 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724190 2639 flags.go:64] FLAG: --runonce="false" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724193 2639 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724196 2639 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724199 2639 flags.go:64] FLAG: --seccomp-default="false" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724202 2639 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724205 2639 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724209 2639 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724212 2639 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724215 2639 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724218 2639 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724221 2639 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724224 2639 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724227 2639 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724230 2639 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724233 2639 flags.go:64] FLAG: --system-cgroups="" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724236 2639 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724242 2639 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724246 2639 flags.go:64] FLAG: --tls-cert-file="" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724250 2639 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724254 2639 flags.go:64] FLAG: --tls-min-version="" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724257 2639 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724260 2639 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 21:15:10.730499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724263 2639 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724267 2639 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724270 2639 flags.go:64] FLAG: --v="2" Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724274 2639 flags.go:64] FLAG: --version="false" Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724279 2639 flags.go:64] FLAG: --vmodule="" Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724283 2639 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724287 2639 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724455 2639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724459 2639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724462 2639 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724465 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724468 2639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724471 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724473 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724476 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724479 2639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724481 2639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724485 2639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724487 2639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724490 2639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724493 2639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:10.731175 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724496 2639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724498 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724501 2639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724504 2639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724509 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724512 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724516 2639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724519 2639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724522 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724525 2639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724527 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724530 2639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724533 2639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724535 2639 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724538 2639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724541 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724543 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724546 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724549 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:10.731713 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724551 2639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724554 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724556 2639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724559 2639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724562 2639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724564 2639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724567 2639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724569 2639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724572 2639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724574 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724577 2639 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724580 2639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724592 2639 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724595 2639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724598 2639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724600 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724603 2639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724607 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724610 2639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724614 2639 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:10.732176 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724617 2639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724619 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724622 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724625 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724627 2639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724630 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724633 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724635 2639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724638 2639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724641 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724643 2639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724646 2639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724648 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724651 2639 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724653 2639 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724656 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724658 2639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724661 2639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724664 2639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:10.732685 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724666 2639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724669 2639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724671 2639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724674 2639 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724677 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724680 2639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724683 2639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724685 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724688 2639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724690 2639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724695 2639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724697 2639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724701 2639 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724704 2639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724706 2639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724709 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724712 2639 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724714 2639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724718 2639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:10.733142 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724721 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724724 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724727 2639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724730 2639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724733 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724736 2639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724738 2639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724741 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724743 2639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724746 2639 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724748 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724751 2639 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724753 2639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724756 2639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724759 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.724789 2639 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:10.733610 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724830 2639 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724834 2639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724838 2639 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724841 2639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724843 2639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724846 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724848 2639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724851 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724855 2639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724860 2639 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724862 2639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724865 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724868 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724870 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724873 2639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724876 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724878 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724881 2639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724884 2639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:10.733991 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724887 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724889 2639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724892 2639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724895 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724898 2639 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724901 2639 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724903 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724906 2639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724909 2639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724912 2639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724914 2639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724917 2639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724919 2639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724922 2639 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724924 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724927 2639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724930 2639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724933 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724936 2639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724938 2639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:10.734538 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724941 2639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724945 2639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724949 2639 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724951 2639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724954 2639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724957 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724959 2639 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724962 2639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724965 2639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724968 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724971 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724974 2639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724977 2639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724979 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724982 2639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724985 2639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724987 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724990 2639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724992 2639 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724995 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:10.735035 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.724998 2639 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725001 2639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725003 2639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725006 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725010 2639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725012 2639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725015 2639 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725018 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725020 2639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725023 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725026 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725028 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725031 2639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725035 2639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725039 2639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725041 2639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725044 2639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725046 2639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725049 2639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725052 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:10.735548 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725054 2639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725058 2639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725062 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725065 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725068 2639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725071 2639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725073 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725076 2639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725078 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725081 2639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725084 2639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725086 2639 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725089 2639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725091 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725095 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725097 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725100 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725103 2639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725105 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:10.736059 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725108 2639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725111 2639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725113 2639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725116 2639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725119 2639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725121 2639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725125 2639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.725129 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.732769 2639 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.732790 2639 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.732807 2639 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.732816 2639 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732879 2639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732884 2639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732887 2639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732890 2639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732893 2639 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732896 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732899 2639 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:10.736533 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732901 2639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732904 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732907 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732910 2639 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732912 2639 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732915 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732918 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732921 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732923 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732926 2639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732928 2639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732931 2639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732934 2639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732936 2639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732939 2639 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732942 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732944 2639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732947 2639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732949 2639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732954 2639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:10.737015 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732960 2639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732963 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732972 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732975 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732977 2639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732980 2639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732983 2639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732986 2639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732989 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732991 2639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732994 2639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732997 2639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.732999 2639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733002 2639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733005 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733007 2639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733010 2639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733012 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733015 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:10.737501 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733018 2639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733020 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733023 2639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733026 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733028 2639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733031 2639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733034 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733036 2639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733039 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733041 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733044 2639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733047 2639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733050 2639 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733053 2639 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733056 2639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733059 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733067 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733069 2639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733072 2639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:10.737964 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733075 2639 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733077 2639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733080 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733082 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733085 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733087 2639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733090 2639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733092 2639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733095 2639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733098 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733100 2639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733103 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733106 2639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733108 2639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733111 2639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733113 2639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733116 2639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733119 2639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733121 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:10.738512 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733124 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733126 2639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733129 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733132 2639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733134 2639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733137 2639 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733140 2639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733144 2639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733148 2639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733152 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733155 2639 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733163 2639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733166 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733169 2639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733171 2639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733174 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733177 2639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733179 2639 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733182 2639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733184 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:10.738969 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733187 2639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733189 2639 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.733220 2639 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733426 2639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733432 2639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733435 2639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733438 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733441 2639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733443 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733446 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733449 2639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733451 2639 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733454 2639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733457 2639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733459 2639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733462 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:10.739471 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733464 2639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733467 2639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733470 2639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733473 2639 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733476 2639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733479 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733481 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733484 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733492 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733495 2639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733497 2639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733500 2639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733503 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733505 2639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733508 2639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733510 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733513 2639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733516 2639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733518 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:10.739867 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733521 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733523 2639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733526 2639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733528 2639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733531 2639 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733534 2639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733536 2639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733539 2639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733541 2639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733544 2639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733546 2639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733549 2639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733551 2639 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733554 2639 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733557 2639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733559 2639 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733563 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733566 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733569 2639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733574 2639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:10.740322 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733577 2639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733580 2639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733588 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733591 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733594 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733597 2639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733600 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733602 2639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733605 2639 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733607 2639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733610 2639 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733613 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733616 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733619 2639 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733622 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733626 2639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733630 2639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733632 2639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733635 2639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:10.740839 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733637 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733640 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733642 2639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733645 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733653 2639 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733656 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733658 2639 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733661 2639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733663 2639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733669 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733672 2639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733674 2639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733677 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733680 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733682 2639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733685 2639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733694 2639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733697 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733700 2639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:10.741506 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733702 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733705 2639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733707 2639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733710 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733712 2639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733715 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733718 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733720 2639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733723 2639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733725 2639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733728 2639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733730 2639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733733 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733735 2639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733738 2639 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733740 2639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:10.742209 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.733765 2639 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733805 2639 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733809 2639 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733812 2639 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733815 2639 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733819 2639 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733823 2639 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733827 2639 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733829 2639 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733832 2639 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733835 2639 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733838 2639 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733840 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733843 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733854 2639 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733858 2639 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733860 2639 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733863 2639 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733865 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733868 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:10.742843 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733870 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733873 2639 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733875 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733878 2639 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733881 2639 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733883 2639 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733886 2639 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733888 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733891 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733893 2639 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733896 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733900 2639 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733904 2639 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733907 2639 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733910 2639 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733914 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733917 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733920 2639 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733924 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:10.743354 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733927 2639 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733930 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733933 2639 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733935 2639 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733938 2639 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733940 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733943 2639 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733946 2639 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733954 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733957 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733960 2639 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733963 2639 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733966 2639 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733969 2639 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733971 2639 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733973 2639 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733976 2639 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733978 2639 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733981 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:10.743829 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733983 2639 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733986 2639 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733989 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733991 2639 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733994 2639 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733997 2639 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.733999 2639 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734002 2639 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734005 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734008 2639 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734010 2639 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734013 2639 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734017 2639 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734020 2639 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734022 2639 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734025 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734027 2639 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734029 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734032 2639 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734035 2639 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:10.744288 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734037 2639 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734040 2639 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734048 2639 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734051 2639 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734053 2639 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734056 2639 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734058 2639 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734061 2639 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734065 2639 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734068 2639 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734070 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734073 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734075 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734078 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734081 2639 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734083 2639 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734086 2639 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734088 2639 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734091 2639 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:10.744834 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734093 2639 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734096 2639 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734099 2639 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734101 2639 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734104 2639 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734108 2639 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734110 2639 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734113 2639 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734116 2639 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:10.734119 2639 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.734207 2639 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.736681 2639 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.737804 2639 server.go:1014] "Starting client certificate rotation" Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.737908 2639 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.738743 2639 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 21:15:10.745286 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.741683 2639 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:10.747016 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.746992 2639 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:10.755945 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.755917 2639 log.go:25] "Validated CRI v1 runtime API" Oct 31 21:15:10.759933 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.759912 2639 log.go:25] "Validated CRI v1 image API" Oct 31 21:15:10.760907 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.760879 2639 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 21:15:10.764099 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.764077 2639 fs.go:135] Filesystem UUIDs: map[20a9b5ca-2a09-4067-b942-bda7fb73d8e9:/dev/nvme0n1p4 73b12867-1cf6-4b74-88e9-7f31b2a4582b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 21:15:10.764155 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.764099 2639 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 21:15:10.770256 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.770121 2639 manager.go:217] Machine: {Timestamp:2026-10-31 21:15:10.768519775 +0000 UTC m=+0.345924045 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099773 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe8ecc0817846daff2017c7427327 SystemUUID:ec2fe8ec-c081-7846-daff-2017c7427327 BootID:96a4953a-69d6-40a0-8e26-08435db256a4 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f8:9a:8a:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f8:9a:8a:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:7d:14:e6:be:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 21:15:10.770256 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.770256 2639 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 21:15:10.770443 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.770362 2639 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 21:15:10.774109 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.774072 2639 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 21:15:10.774291 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.774111 2639 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 21:15:10.774342 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.774293 2639 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 21:15:10.774342 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.774302 2639 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 21:15:10.774342 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.774329 2639 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 21:15:10.774438 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.774343 2639 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 21:15:10.775569 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.775558 2639 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 21:15:10.780121 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.780097 2639 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 21:15:10.780273 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.780263 2639 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 21:15:10.781205 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.781194 2639 kubelet.go:493] "Attempting to sync node with API server" Oct 31 21:15:10.781250 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.781215 2639 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 21:15:10.781250 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.781226 2639 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 21:15:10.781250 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.781236 2639 kubelet.go:405] "Adding apiserver pod source" Oct 31 21:15:10.781250 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.781246 2639 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 21:15:10.781747 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.781729 2639 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 21:15:10.784773 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.784732 2639 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 21:15:10.792863 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.792838 2639 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 21:15:10.793750 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.793736 2639 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 21:15:10.793797 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.793768 2639 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 21:15:10.793938 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.793927 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 21:15:10.793972 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.793944 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 21:15:10.793972 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.793950 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 21:15:10.793972 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.793956 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 21:15:10.793972 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.793962 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 21:15:10.793972 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.793968 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 21:15:10.794662 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.794653 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 21:15:10.794692 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.794662 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 21:15:10.794692 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.794671 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 21:15:10.794692 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.794676 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 21:15:10.794692 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.794686 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 21:15:10.794817 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.794695 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 21:15:10.795386 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.795378 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 21:15:10.795429 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.795387 2639 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 21:15:10.799587 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.799558 2639 server.go:1263] "Started kubelet" Oct 31 21:15:10.799755 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.799700 2639 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 21:15:10.799834 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.799724 2639 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 21:15:10.799834 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.799799 2639 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 21:15:10.800531 ip-10-0-131-21 systemd[1]: Started Kubernetes Kubelet. Oct 31 21:15:10.800735 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.800723 2639 server.go:317] "Adding debug handlers to kubelet server" Oct 31 21:15:10.801599 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.801511 2639 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 21:15:10.806378 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.806357 2639 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 21:15:10.808425 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.808404 2639 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 21:15:10.809696 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.809679 2639 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 21:15:10.809902 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.809882 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:10.810499 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.810480 2639 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 21:15:10.810616 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.810489 2639 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 21:15:10.810616 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.810619 2639 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 21:15:10.810832 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.810817 2639 reconstruct.go:97] "Volume reconstruction finished" Oct 31 21:15:10.810867 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.810835 2639 reconciler.go:29] "Reconciler: start to sync state" Oct 31 21:15:10.811250 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.811236 2639 factory.go:55] Registering systemd factory Oct 31 21:15:10.811298 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.811253 2639 factory.go:223] Registration of the systemd container factory successfully Oct 31 21:15:10.811585 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.811568 2639 factory.go:153] Registering CRI-O factory Oct 31 21:15:10.811585 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.811587 2639 factory.go:223] Registration of the crio container factory successfully Oct 31 21:15:10.811722 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.811638 2639 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 21:15:10.811722 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.811655 2639 factory.go:103] Registering Raw factory Oct 31 21:15:10.811722 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.811667 2639 manager.go:1199] Started watching for new ooms in manager Oct 31 21:15:10.812963 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.812942 2639 manager.go:319] Starting recovery of all containers Oct 31 21:15:10.825921 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.825896 2639 manager.go:324] Recovery completed Oct 31 21:15:10.830840 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.830698 2639 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 21:15:10.832134 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.832114 2639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:10.832257 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.832142 2639 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-21.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 21:15:10.834277 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.834255 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:10.834354 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.834287 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:10.834354 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.834298 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:10.834840 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.834825 2639 cpu_manager.go:226] "Starting" policy="none" Oct 31 21:15:10.834840 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.834838 2639 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 21:15:10.834945 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.834845 2639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:10.834945 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.834858 2639 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 21:15:10.835525 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.835512 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:10.835562 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.835537 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:10.835562 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.835548 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:10.837100 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.837087 2639 policy_none.go:50] "Start" Oct 31 21:15:10.837148 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.837104 2639 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 21:15:10.837148 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.837114 2639 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 21:15:10.839142 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.839125 2639 policy_none.go:44] "Start" Oct 31 21:15:10.842588 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.841327 2639 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-21.ec2.internal.18e3b9bff981f03b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-21.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-21.ec2.internal,},FirstTimestamp:2026-10-31 21:15:10.799523899 +0000 UTC m=+0.376928168,LastTimestamp:2026-10-31 21:15:10.799523899 +0000 UTC m=+0.376928168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-21.ec2.internal,}" Oct 31 21:15:10.853072 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.852705 2639 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 21:15:10.853072 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.852714 2639 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 21:15:10.853072 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.852888 2639 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 21:15:10.853072 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.852940 2639 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-21.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 21:15:10.861670 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.861641 2639 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7bs4r" Oct 31 21:15:10.871922 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.871892 2639 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7bs4r" Oct 31 21:15:10.873652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.873604 2639 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.884001 2639 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-21.ec2.internal.18e3b9bffb942f2b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-21.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-21.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-21.ec2.internal,},FirstTimestamp:2026-10-31 21:15:10.834274091 +0000 UTC m=+0.411678362,LastTimestamp:2026-10-31 21:15:10.834274091 +0000 UTC m=+0.411678362,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-21.ec2.internal,}" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.887526 2639 manager.go:341] "Starting Device Plugin manager" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.888228 2639 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.888247 2639 server.go:87] "Starting device plugin registration server" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.888490 2639 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.888500 2639 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.888659 2639 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.888751 2639 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.888762 2639 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.889360 2639 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.889469 2639 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.893013 2639 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.894269 2639 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.894296 2639 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.894319 2639 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 21:15:10.896691 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:10.894359 2639 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 21:15:10.910557 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.910542 2639 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:10.989633 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.989549 2639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:10.991118 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.991098 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:10.991227 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.991131 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:10.991227 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.991142 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:10.991227 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.991181 2639 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-21.ec2.internal" Oct 31 21:15:10.995339 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.995315 2639 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal"] Oct 31 21:15:10.995414 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.995384 2639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:10.996541 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.996520 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:10.996629 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.996549 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:10.996629 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.996559 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:10.998725 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.998711 2639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:10.998895 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.998881 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:10.998935 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:10.998917 2639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:11.000112 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.000093 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:11.000220 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.000127 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:11.000220 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.000138 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:11.000220 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.000099 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:11.000220 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.000190 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:11.000220 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.000208 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:11.002326 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.002309 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.002445 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.002338 2639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:11.003109 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.003092 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:11.003214 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.003120 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:11.003214 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.003155 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:11.009667 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.009645 2639 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.009766 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:11.009675 2639 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-21.ec2.internal\": node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:11.011832 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.011812 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.011899 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.011842 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.011899 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.011858 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d208acf97ac3e1709a838235c7640507-config\") pod \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" (UID: \"d208acf97ac3e1709a838235c7640507\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.038441 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:11.038421 2639 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-21.ec2.internal\" not found" node="ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.040383 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:11.040363 2639 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-21.ec2.internal\" not found" node="ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.047704 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:11.047686 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:11.112603 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.112565 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.112603 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.112603 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.112719 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.112623 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d208acf97ac3e1709a838235c7640507-config\") pod \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" (UID: \"d208acf97ac3e1709a838235c7640507\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.112719 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.112654 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d208acf97ac3e1709a838235c7640507-config\") pod \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" (UID: \"d208acf97ac3e1709a838235c7640507\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.112719 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.112660 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.112719 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:11.112660 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:11.148498 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:11.148456 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:11.248987 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:11.248919 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:12.458611 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:12.458574 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:12.460723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:12.460704 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:12.464877 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:12.464857 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:12.565416 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:12.565370 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:12.665960 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:12.665876 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:12.766490 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:12.766451 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:12.852952 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:12.852915 2639 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 21:15:12.853594 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:12.853077 2639 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 21:15:12.866898 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:12.866868 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:12.909701 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:12.909667 2639 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:12.967820 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:12.967748 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:12.998288 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:12.998236 2639 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 21:10:11 +0000 UTC" deadline="2028-05-19 18:10:45.35767522 +0000 UTC" Oct 31 21:15:12.998288 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:12.998277 2639 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13580h55m32.359402132s" Oct 31 21:15:13.037880 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.037833 2639 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:13.039149 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.039127 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:13.039276 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.039164 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:13.039276 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.039177 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:13.068859 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:13.068825 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:13.150732 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:13.150690 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76456f0ae0dd752db0186d1d3e70e3aa.slice/crio-24ad4aaa02fcb3c760a3fe8813664d0215626be148def102ca59e19d7e3308f5 WatchSource:0}: Error finding container 24ad4aaa02fcb3c760a3fe8813664d0215626be148def102ca59e19d7e3308f5: Status 404 returned error can't find the container with id 24ad4aaa02fcb3c760a3fe8813664d0215626be148def102ca59e19d7e3308f5 Oct 31 21:15:13.150989 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:13.150968 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd208acf97ac3e1709a838235c7640507.slice/crio-4397200e77b1c2a7f7a1a3354f9066a6a7193afa7d85f8ab9f5c97e84ebedf11 WatchSource:0}: Error finding container 4397200e77b1c2a7f7a1a3354f9066a6a7193afa7d85f8ab9f5c97e84ebedf11: Status 404 returned error can't find the container with id 4397200e77b1c2a7f7a1a3354f9066a6a7193afa7d85f8ab9f5c97e84ebedf11 Oct 31 21:15:13.156993 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.156978 2639 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:15:13.168964 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:13.168943 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:13.217200 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.217172 2639 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:13.269621 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:13.269553 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:13.370255 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:13.370222 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:13.470930 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:13.470888 2639 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:13.492366 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.492315 2639 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:13.525379 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.525283 2639 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:13.551833 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.551802 2639 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:15:13.553744 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.552950 2639 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:13.573956 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.573930 2639 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:15:13.899408 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.897716 2639 apiserver.go:52] "Watching apiserver" Oct 31 21:15:13.924525 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.923803 2639 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 21:15:13.938760 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.938728 2639 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 21:15:13.939048 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.939031 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-whzgx","kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mrskg","openshift-multus/multus-additional-cni-plugins-2l8b7","openshift-network-operator/iptables-alerter-x47n8","openshift-multus/multus-jxbcm","openshift-multus/network-metrics-daemon-zc54c","kube-system/konnectivity-agent-9kbsn","openshift-image-registry/node-ca-42vxn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg","openshift-dns/node-resolver-vx79w","openshift-network-diagnostics/network-check-target-tpl7t","openshift-ovn-kubernetes/ovnkube-node-8nl7t"] Oct 31 21:15:13.941962 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.941635 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:13.941962 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:13.941740 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:13.943985 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.943863 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:13.943985 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:13.943944 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:13.946169 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.946137 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:13.948425 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.948406 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:13.950882 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.950861 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:13.954051 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.953796 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxbcm" Oct 31 21:15:13.956346 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.956292 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:13.958880 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.958850 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:13.958993 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:13.958932 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:13.961264 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.961239 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:13.961550 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.961533 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:13.961645 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.961572 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:13.961700 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.961673 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6mmhq\"" Oct 31 21:15:13.963115 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.963089 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 21:15:13.963237 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.963136 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 21:15:13.963237 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.963198 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xsvd6\"" Oct 31 21:15:13.963363 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.963249 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 21:15:13.963363 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.963276 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 21:15:13.963913 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.963893 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:13.964351 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.964314 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 21:15:13.965118 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.965099 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:13.965482 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.965421 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:13.965482 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.965440 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 21:15:13.965482 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.965482 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9gmld\"" Oct 31 21:15:13.966899 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.966881 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:13.969377 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.969049 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:13.969377 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.969227 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p85m4\"" Oct 31 21:15:13.969991 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.969971 2639 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 21:15:13.970083 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.970045 2639 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8pjlv" Oct 31 21:15:13.982121 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:13.981510 2639 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8pjlv" Oct 31 21:15:14.014291 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.014232 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" event={"ID":"76456f0ae0dd752db0186d1d3e70e3aa","Type":"ContainerStarted","Data":"24ad4aaa02fcb3c760a3fe8813664d0215626be148def102ca59e19d7e3308f5"} Oct 31 21:15:14.015871 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.015838 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" event={"ID":"d208acf97ac3e1709a838235c7640507","Type":"ContainerStarted","Data":"4397200e77b1c2a7f7a1a3354f9066a6a7193afa7d85f8ab9f5c97e84ebedf11"} Oct 31 21:15:14.027016 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.026992 2639 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 21:15:14.037209 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037176 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-socket-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.037400 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037222 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725ms\" (UniqueName: \"kubernetes.io/projected/626d59af-b777-449a-87cc-23e763dd495f-kube-api-access-725ms\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.037400 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037387 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-cni-binary-copy\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.037518 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037424 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-netns\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.037518 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037456 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhwr\" (UniqueName: \"kubernetes.io/projected/47354b4e-2a35-40be-a539-07dad787bd97-kube-api-access-8jhwr\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.037619 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037542 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-run\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.037619 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037581 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.037619 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037608 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-config\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.037759 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037656 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-registration-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.037759 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037708 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-sys-fs\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.037759 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037736 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-iptables-alerter-script\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.037884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037773 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-host-slash\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.037884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037825 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpsd4\" (UniqueName: \"kubernetes.io/projected/6805be97-0a2e-490c-a50c-6700f15c4abc-kube-api-access-vpsd4\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.037884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037849 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-sys\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.038000 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037873 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-slash\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.038000 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037944 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.038000 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037979 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:14.038000 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.037996 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpcxl\" (UniqueName: \"kubernetes.io/projected/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-kube-api-access-hpcxl\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.038136 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038025 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-socket-dir-parent\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.038136 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038053 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-multus-daemon-config\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.038136 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038079 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-kubelet-config\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:14.038282 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038171 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-etc-kubernetes\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.038282 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038225 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-systemd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.038282 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038251 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.038474 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038313 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-k8s-cni-cncf-io\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.038474 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038366 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:14.038474 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038387 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-host\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.038474 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038407 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455ft\" (UniqueName: \"kubernetes.io/projected/672a2ce2-d45d-4f0c-a75a-b018105640aa-kube-api-access-455ft\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.038474 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038457 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-script-lib\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.038707 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038552 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-modprobe-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.038707 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038587 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/672a2ce2-d45d-4f0c-a75a-b018105640aa-serviceca\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.038707 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038613 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-var-lib-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.038707 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038638 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-node-log\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.038707 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038669 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-netd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.038922 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038709 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-env-overrides\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.038922 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038741 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.038922 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038763 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-bin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.038922 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038816 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-system-cni-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.038922 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038888 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6805be97-0a2e-490c-a50c-6700f15c4abc-tmp-dir\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.039160 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038937 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58tf\" (UniqueName: \"kubernetes.io/projected/3616b3de-b6e6-497f-8546-8fef766c309f-kube-api-access-f58tf\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:14.039160 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038968 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-tmp\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.039160 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.038995 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.039160 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039022 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-log-socket\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.039160 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039044 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-dbus\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:14.039160 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039069 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysconfig\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.039160 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039095 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfgnv\" (UniqueName: \"kubernetes.io/projected/a4772ac2-3076-4099-89da-cb3be3d4387c-kube-api-access-tfgnv\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.039160 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039118 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/672a2ce2-d45d-4f0c-a75a-b018105640aa-host\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.039160 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039150 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcwz\" (UniqueName: \"kubernetes.io/projected/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-kube-api-access-bjcwz\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039173 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-device-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039198 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-system-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039225 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-hostroot\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039249 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039274 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039299 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-conf\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039342 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039372 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-ovn\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039397 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039441 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac6429fb-48a7-4691-9079-4ed168f1da32-agent-certs\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039465 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac6429fb-48a7-4691-9079-4ed168f1da32-konnectivity-ca\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039489 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t57pd\" (UniqueName: \"kubernetes.io/projected/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-kube-api-access-t57pd\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039520 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039544 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-lib-modules\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039568 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-tuned\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.039652 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039603 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-systemd-units\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039625 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-cnibin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039652 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-kubernetes\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039675 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-systemd\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039700 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-var-lib-kubelet\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039722 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-kubelet\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039746 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-bin\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039770 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039793 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-multus\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039822 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-conf-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039853 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039877 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-netns\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039900 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-kubelet\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039931 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6805be97-0a2e-490c-a50c-6700f15c4abc-hosts-file\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039954 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-etc-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.039978 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.040001 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-multus-certs\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.040411 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.040037 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cnibin\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.064643 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.064557 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 21:15:14.064643 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.064557 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 21:15:14.064643 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.064564 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 21:15:14.064643 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.064557 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 21:15:14.064994 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.064777 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 21:15:14.064994 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.064892 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 21:15:14.064994 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.064915 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 21:15:14.064994 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.064974 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdqcm\"" Oct 31 21:15:14.064994 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.064999 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 21:15:14.065231 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065030 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vddbx\"" Oct 31 21:15:14.065231 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065051 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 21:15:14.065231 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065076 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ddmwn\"" Oct 31 21:15:14.065231 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065092 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 21:15:14.065231 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065113 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 21:15:14.065231 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065130 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 21:15:14.065231 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065145 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-rdjnn\"" Oct 31 21:15:14.065231 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065016 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 21:15:14.065231 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065182 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 21:15:14.065534 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065237 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 21:15:14.065534 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065245 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 21:15:14.065534 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065348 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g9h45\"" Oct 31 21:15:14.065534 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.065432 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 21:15:14.140445 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140387 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-systemd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.140445 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140445 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140487 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-k8s-cni-cncf-io\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140518 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140542 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-host\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140561 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140567 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-systemd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140566 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455ft\" (UniqueName: \"kubernetes.io/projected/672a2ce2-d45d-4f0c-a75a-b018105640aa-kube-api-access-455ft\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140632 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-host\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140640 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-script-lib\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140669 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-modprobe-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.140690 2639 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140698 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/672a2ce2-d45d-4f0c-a75a-b018105640aa-serviceca\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140567 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-k8s-cni-cncf-io\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.140705 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140714 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-var-lib-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140734 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-node-log\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.140767 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:14.640743519 +0000 UTC m=+3.103218325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140788 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-netd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140818 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-env-overrides\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140835 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-modprobe-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140845 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140870 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-bin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140895 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-system-cni-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140924 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6805be97-0a2e-490c-a50c-6700f15c4abc-tmp-dir\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140947 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58tf\" (UniqueName: \"kubernetes.io/projected/3616b3de-b6e6-497f-8546-8fef766c309f-kube-api-access-f58tf\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140953 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-var-lib-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.140965 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-tmp\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141010 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141040 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-log-socket\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141063 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-dbus\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:14.141125 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141088 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysconfig\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141112 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgnv\" (UniqueName: \"kubernetes.io/projected/a4772ac2-3076-4099-89da-cb3be3d4387c-kube-api-access-tfgnv\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141136 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/672a2ce2-d45d-4f0c-a75a-b018105640aa-host\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141160 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcwz\" (UniqueName: \"kubernetes.io/projected/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-kube-api-access-bjcwz\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141185 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-device-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141187 2639 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141194 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/672a2ce2-d45d-4f0c-a75a-b018105640aa-serviceca\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141211 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-system-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141235 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-hostroot\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141243 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-node-log\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141261 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141287 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141313 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-conf\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141342 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-dbus\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141355 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141365 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-script-lib\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141395 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-ovn\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141406 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-log-socket\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.141850 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141433 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141463 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac6429fb-48a7-4691-9079-4ed168f1da32-agent-certs\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141479 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysconfig\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141467 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-hostroot\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141496 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac6429fb-48a7-4691-9079-4ed168f1da32-konnectivity-ca\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141534 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57pd\" (UniqueName: \"kubernetes.io/projected/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-kube-api-access-t57pd\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141558 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141581 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-lib-modules\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141608 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-tuned\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141631 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-systemd-units\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141654 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-cnibin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141675 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-kubernetes\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141708 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-systemd\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141730 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-var-lib-kubelet\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141737 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141761 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-kubelet\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141766 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/672a2ce2-d45d-4f0c-a75a-b018105640aa-host\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141787 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-bin\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.142824 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141808 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141831 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-multus\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141853 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-conf-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141882 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141909 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141911 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-netns\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141945 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-netns\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141960 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-kubelet\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141993 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6805be97-0a2e-490c-a50c-6700f15c4abc-hosts-file\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141997 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-kubelet\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142017 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-etc-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142025 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac6429fb-48a7-4691-9079-4ed168f1da32-konnectivity-ca\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142041 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142079 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-multus-certs\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142104 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142107 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cnibin\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142128 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-device-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.143688 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142122 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-conf\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142143 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cnibin\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142152 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-socket-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142179 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725ms\" (UniqueName: \"kubernetes.io/projected/626d59af-b777-449a-87cc-23e763dd495f-kube-api-access-725ms\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142205 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-cni-binary-copy\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142232 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-netns\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142235 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142256 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhwr\" (UniqueName: \"kubernetes.io/projected/47354b4e-2a35-40be-a539-07dad787bd97-kube-api-access-8jhwr\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142298 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-ovn\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142341 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-run\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142370 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142395 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-config\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142418 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-registration-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142443 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-sys-fs\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142471 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-iptables-alerter-script\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142489 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142499 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-host-slash\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.144483 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142526 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-env-overrides\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142537 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpsd4\" (UniqueName: \"kubernetes.io/projected/6805be97-0a2e-490c-a50c-6700f15c4abc-kube-api-access-vpsd4\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142540 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142589 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-multus-certs\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141436 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-system-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142869 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-system-cni-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.142921 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-bin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143147 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-run\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143205 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143444 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-sys\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143477 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-slash\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143502 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143539 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143540 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-sys-fs\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143565 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcxl\" (UniqueName: \"kubernetes.io/projected/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-kube-api-access-hpcxl\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143592 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-socket-dir-parent\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143597 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-registration-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.145225 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143617 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-multus-daemon-config\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143644 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-kubelet-config\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143672 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-etc-kubernetes\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143694 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143779 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-etc-kubernetes\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.143955 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-socket-dir-parent\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144072 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144140 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-systemd-units\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144198 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-cnibin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144249 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-kubernetes\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144297 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-systemd\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144360 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-var-lib-kubelet\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144402 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-kubelet\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144453 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-bin\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144495 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-multus-daemon-config\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144509 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144532 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-iptables-alerter-script\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144541 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-kubelet-config\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:14.146028 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144551 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-multus\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144572 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-slash\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144592 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-conf-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144602 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-sys\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144707 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-socket-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.141885 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-lib-modules\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.144795 2639 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.144853 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret podName:72751047-bc3c-4f2d-9655-e23d5211d346 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:14.644835198 +0000 UTC m=+3.107309988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret") pod "global-pull-secret-syncer-whzgx" (UID: "72751047-bc3c-4f2d-9655-e23d5211d346") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144904 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6805be97-0a2e-490c-a50c-6700f15c4abc-hosts-file\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144908 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-cni-binary-copy\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144945 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-etc-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.144963 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-netd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.145001 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-netns\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.145036 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-host-slash\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.145355 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6805be97-0a2e-490c-a50c-6700f15c4abc-tmp-dir\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.145877 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-tmp\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.146853 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.145916 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-config\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.147968 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.147936 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.148108 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.148042 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac6429fb-48a7-4691-9079-4ed168f1da32-agent-certs\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:14.148226 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.148197 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-tuned\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.165277 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.165127 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-455ft\" (UniqueName: \"kubernetes.io/projected/672a2ce2-d45d-4f0c-a75a-b018105640aa-kube-api-access-455ft\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.167122 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.167096 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57pd\" (UniqueName: \"kubernetes.io/projected/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-kube-api-access-t57pd\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.167668 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.167361 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:14.167668 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.167386 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:14.167668 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.167402 2639 projected.go:196] Error preparing data for projected volume kube-api-access-fc8sb for pod openshift-network-diagnostics/network-check-target-tpl7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:14.167668 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.167484 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb podName:c929e38e-147c-41f9-ad34-b0af48044e09 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:14.667457964 +0000 UTC m=+3.129932769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fc8sb" (UniqueName: "kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb") pod "network-check-target-tpl7t" (UID: "c929e38e-147c-41f9-ad34-b0af48044e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:14.167668 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.167626 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcwz\" (UniqueName: \"kubernetes.io/projected/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-kube-api-access-bjcwz\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.167668 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.167651 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgnv\" (UniqueName: \"kubernetes.io/projected/a4772ac2-3076-4099-89da-cb3be3d4387c-kube-api-access-tfgnv\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.171278 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.171152 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58tf\" (UniqueName: \"kubernetes.io/projected/3616b3de-b6e6-497f-8546-8fef766c309f-kube-api-access-f58tf\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:14.171278 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.171241 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-725ms\" (UniqueName: \"kubernetes.io/projected/626d59af-b777-449a-87cc-23e763dd495f-kube-api-access-725ms\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.171546 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.171525 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcxl\" (UniqueName: \"kubernetes.io/projected/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-kube-api-access-hpcxl\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.171624 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.171592 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhwr\" (UniqueName: \"kubernetes.io/projected/47354b4e-2a35-40be-a539-07dad787bd97-kube-api-access-8jhwr\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.171787 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.171767 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpsd4\" (UniqueName: \"kubernetes.io/projected/6805be97-0a2e-490c-a50c-6700f15c4abc-kube-api-access-vpsd4\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.236683 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.236647 2639 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:14.260354 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.260278 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:14.283435 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.283398 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:14.289843 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.289815 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:14.294079 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.294044 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxbcm" Oct 31 21:15:14.294603 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:14.294575 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bb99e2_7626_47ee_b7b6_6b45e954cf4a.slice/crio-c483d73912b3bed0811ff9aa7cc9ae3e170f7865023d4c8c843a20c766f7fec3 WatchSource:0}: Error finding container c483d73912b3bed0811ff9aa7cc9ae3e170f7865023d4c8c843a20c766f7fec3: Status 404 returned error can't find the container with id c483d73912b3bed0811ff9aa7cc9ae3e170f7865023d4c8c843a20c766f7fec3 Oct 31 21:15:14.302142 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.302116 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:14.302436 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:14.302411 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bf6948_d5fa_4e37_8f85_d2d9dd9d4936.slice/crio-67f935e7f23e653d91ae48b9757785164fe436dfcb3cfd1aa3ac377e5523a6f7 WatchSource:0}: Error finding container 67f935e7f23e653d91ae48b9757785164fe436dfcb3cfd1aa3ac377e5523a6f7: Status 404 returned error can't find the container with id 67f935e7f23e653d91ae48b9757785164fe436dfcb3cfd1aa3ac377e5523a6f7 Oct 31 21:15:14.308267 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:14.308243 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47354b4e_2a35_40be_a539_07dad787bd97.slice/crio-6b41925f038f7f1b008e0dc442614db444df41bcc0e657bcba4b7247f54cb1cf WatchSource:0}: Error finding container 6b41925f038f7f1b008e0dc442614db444df41bcc0e657bcba4b7247f54cb1cf: Status 404 returned error can't find the container with id 6b41925f038f7f1b008e0dc442614db444df41bcc0e657bcba4b7247f54cb1cf Oct 31 21:15:14.315188 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:14.315016 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672a2ce2_d45d_4f0c_a75a_b018105640aa.slice/crio-8b9c15d63056b516488b4f8bd97f8abfe7a9b2462a440559ec4553afe3dcdf6c WatchSource:0}: Error finding container 8b9c15d63056b516488b4f8bd97f8abfe7a9b2462a440559ec4553afe3dcdf6c: Status 404 returned error can't find the container with id 8b9c15d63056b516488b4f8bd97f8abfe7a9b2462a440559ec4553afe3dcdf6c Oct 31 21:15:14.325561 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.325535 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:14.329562 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.329534 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:14.336505 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.336467 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:14.337095 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:14.337060 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6429fb_48a7_4691_9079_4ed168f1da32.slice/crio-6cb8261c54eb4d093fb57b35866eeed8890e0e79f0aafa1f4ef72800cf166673 WatchSource:0}: Error finding container 6cb8261c54eb4d093fb57b35866eeed8890e0e79f0aafa1f4ef72800cf166673: Status 404 returned error can't find the container with id 6cb8261c54eb4d093fb57b35866eeed8890e0e79f0aafa1f4ef72800cf166673 Oct 31 21:15:14.337996 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.337943 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:14.340657 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:14.340630 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb10a24_4ad9_48b0_8f68_3f2fd59d07d6.slice/crio-1e5b3d4df07a02a288eb471c3cf70a561cf4f36ebefb80f5ea0f14eee32e066d WatchSource:0}: Error finding container 1e5b3d4df07a02a288eb471c3cf70a561cf4f36ebefb80f5ea0f14eee32e066d: Status 404 returned error can't find the container with id 1e5b3d4df07a02a288eb471c3cf70a561cf4f36ebefb80f5ea0f14eee32e066d Oct 31 21:15:14.350281 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:14.349890 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626d59af_b777_449a_87cc_23e763dd495f.slice/crio-387844b07ce349c89d99b6c1db13f3c084e549bc3a56626339033905e54ffc94 WatchSource:0}: Error finding container 387844b07ce349c89d99b6c1db13f3c084e549bc3a56626339033905e54ffc94: Status 404 returned error can't find the container with id 387844b07ce349c89d99b6c1db13f3c084e549bc3a56626339033905e54ffc94 Oct 31 21:15:14.351760 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:14.351694 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6805be97_0a2e_490c_a50c_6700f15c4abc.slice/crio-d2849f1ebd647fe71e6d119c469c77e22cc3b8ce434a3832c1622ec52aaa26c7 WatchSource:0}: Error finding container d2849f1ebd647fe71e6d119c469c77e22cc3b8ce434a3832c1622ec52aaa26c7: Status 404 returned error can't find the container with id d2849f1ebd647fe71e6d119c469c77e22cc3b8ce434a3832c1622ec52aaa26c7 Oct 31 21:15:14.647097 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.647058 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:14.647233 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.647115 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:14.647233 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.647216 2639 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:14.647356 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.647233 2639 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:14.647356 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.647286 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:15.647266669 +0000 UTC m=+4.109741475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:14.647356 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.647303 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret podName:72751047-bc3c-4f2d-9655-e23d5211d346 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:15.64729565 +0000 UTC m=+4.109770443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret") pod "global-pull-secret-syncer-whzgx" (UID: "72751047-bc3c-4f2d-9655-e23d5211d346") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:14.748744 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.748480 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:14.748912 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.748705 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:14.748912 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.748876 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:14.748912 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.748894 2639 projected.go:196] Error preparing data for projected volume kube-api-access-fc8sb for pod openshift-network-diagnostics/network-check-target-tpl7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:14.749021 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:14.748958 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb podName:c929e38e-147c-41f9-ad34-b0af48044e09 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:15.748939642 +0000 UTC m=+4.211414433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fc8sb" (UniqueName: "kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb") pod "network-check-target-tpl7t" (UID: "c929e38e-147c-41f9-ad34-b0af48044e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:14.984590 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.984477 2639 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 21:10:13 +0000 UTC" deadline="2028-06-01 21:31:03.892503138 +0000 UTC" Oct 31 21:15:14.984590 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:14.984540 2639 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13896h15m48.907967541s" Oct 31 21:15:15.010435 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.010404 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:15.010594 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:15.010553 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:15.022151 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.022109 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrskg" event={"ID":"a4772ac2-3076-4099-89da-cb3be3d4387c","Type":"ContainerStarted","Data":"00fe079374db9248b33233fb9dd4d6f0086a037960471fb0982c5d71100adf47"} Oct 31 21:15:15.026597 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.025591 2639 generic.go:356] "Generic (PLEG): container finished" podID="76456f0ae0dd752db0186d1d3e70e3aa" containerID="ad29c852c4ea2d970962908954294f93db2665e7f36812d7c047ab5b0de30e58" exitCode=0 Oct 31 21:15:15.026597 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.025686 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" event={"ID":"76456f0ae0dd752db0186d1d3e70e3aa","Type":"ContainerDied","Data":"ad29c852c4ea2d970962908954294f93db2665e7f36812d7c047ab5b0de30e58"} Oct 31 21:15:15.038547 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.038420 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vx79w" event={"ID":"6805be97-0a2e-490c-a50c-6700f15c4abc","Type":"ContainerStarted","Data":"d2849f1ebd647fe71e6d119c469c77e22cc3b8ce434a3832c1622ec52aaa26c7"} Oct 31 21:15:15.055786 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.055627 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" event={"ID":"626d59af-b777-449a-87cc-23e763dd495f","Type":"ContainerStarted","Data":"387844b07ce349c89d99b6c1db13f3c084e549bc3a56626339033905e54ffc94"} Oct 31 21:15:15.068421 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.068381 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" event={"ID":"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6","Type":"ContainerStarted","Data":"1e5b3d4df07a02a288eb471c3cf70a561cf4f36ebefb80f5ea0f14eee32e066d"} Oct 31 21:15:15.091926 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.091882 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9kbsn" event={"ID":"ac6429fb-48a7-4691-9079-4ed168f1da32","Type":"ContainerStarted","Data":"6cb8261c54eb4d093fb57b35866eeed8890e0e79f0aafa1f4ef72800cf166673"} Oct 31 21:15:15.110180 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.110095 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" event={"ID":"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a","Type":"ContainerStarted","Data":"c483d73912b3bed0811ff9aa7cc9ae3e170f7865023d4c8c843a20c766f7fec3"} Oct 31 21:15:15.117164 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.117084 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" event={"ID":"d208acf97ac3e1709a838235c7640507","Type":"ContainerStarted","Data":"91d8b0f6d2263de09dde978a9718c1e7d9ccdab329624d7e8273aaab4e475d9a"} Oct 31 21:15:15.135978 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.135939 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-42vxn" event={"ID":"672a2ce2-d45d-4f0c-a75a-b018105640aa","Type":"ContainerStarted","Data":"8b9c15d63056b516488b4f8bd97f8abfe7a9b2462a440559ec4553afe3dcdf6c"} Oct 31 21:15:15.156634 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.156567 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" podStartSLOduration=2.156547029 podStartE2EDuration="2.156547029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:15.155530662 +0000 UTC m=+3.618005477" watchObservedRunningTime="2026-10-31 21:15:15.156547029 +0000 UTC m=+3.619021843" Oct 31 21:15:15.160214 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.160175 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxbcm" event={"ID":"47354b4e-2a35-40be-a539-07dad787bd97","Type":"ContainerStarted","Data":"6b41925f038f7f1b008e0dc442614db444df41bcc0e657bcba4b7247f54cb1cf"} Oct 31 21:15:15.167481 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.167436 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x47n8" event={"ID":"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936","Type":"ContainerStarted","Data":"67f935e7f23e653d91ae48b9757785164fe436dfcb3cfd1aa3ac377e5523a6f7"} Oct 31 21:15:15.656914 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.656818 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:15.656914 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.656862 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:15.657215 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:15.656955 2639 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:15.657215 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:15.657017 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret podName:72751047-bc3c-4f2d-9655-e23d5211d346 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:17.657000377 +0000 UTC m=+6.119475176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret") pod "global-pull-secret-syncer-whzgx" (UID: "72751047-bc3c-4f2d-9655-e23d5211d346") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:15.657488 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:15.657439 2639 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:15.657696 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:15.657498 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:17.657480979 +0000 UTC m=+6.119955775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:15.758562 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.757908 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:15.758562 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:15.758109 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:15.758562 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:15.758131 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:15.758562 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:15.758146 2639 projected.go:196] Error preparing data for projected volume kube-api-access-fc8sb for pod openshift-network-diagnostics/network-check-target-tpl7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:15.758562 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:15.758210 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb podName:c929e38e-147c-41f9-ad34-b0af48044e09 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:17.758190059 +0000 UTC m=+6.220664872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fc8sb" (UniqueName: "kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb") pod "network-check-target-tpl7t" (UID: "c929e38e-147c-41f9-ad34-b0af48044e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:15.985260 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.985167 2639 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 21:10:13 +0000 UTC" deadline="2028-06-17 01:28:55.01040557 +0000 UTC" Oct 31 21:15:15.985260 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:15.985212 2639 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14260h13m39.025198087s" Oct 31 21:15:16.011532 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:16.011464 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:16.011532 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:16.011512 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:16.011747 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:16.011605 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:16.011747 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:16.011711 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:16.179221 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:16.179184 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" event={"ID":"76456f0ae0dd752db0186d1d3e70e3aa","Type":"ContainerStarted","Data":"b35f576ff1f943ae20965f9794d295ae98788eaed83a95b9db9e18a5ba92ef87"} Oct 31 21:15:16.223741 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:16.223671 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" podStartSLOduration=3.223651562 podStartE2EDuration="3.223651562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:16.223192273 +0000 UTC m=+4.685667086" watchObservedRunningTime="2026-10-31 21:15:16.223651562 +0000 UTC m=+4.686126377" Oct 31 21:15:17.010343 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:17.010268 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:17.010795 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:17.010436 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:17.676434 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:17.676348 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:17.676592 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:17.676452 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:17.676592 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:17.676569 2639 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:17.676693 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:17.676638 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:21.676615649 +0000 UTC m=+10.139090464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:17.677054 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:17.677032 2639 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:17.677161 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:17.677087 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret podName:72751047-bc3c-4f2d-9655-e23d5211d346 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:21.677072118 +0000 UTC m=+10.139546910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret") pod "global-pull-secret-syncer-whzgx" (UID: "72751047-bc3c-4f2d-9655-e23d5211d346") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:17.777585 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:17.777547 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:17.777767 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:17.777743 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:17.777767 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:17.777766 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:17.777879 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:17.777779 2639 projected.go:196] Error preparing data for projected volume kube-api-access-fc8sb for pod openshift-network-diagnostics/network-check-target-tpl7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:17.777879 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:17.777838 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb podName:c929e38e-147c-41f9-ad34-b0af48044e09 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:21.777820195 +0000 UTC m=+10.240295005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fc8sb" (UniqueName: "kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb") pod "network-check-target-tpl7t" (UID: "c929e38e-147c-41f9-ad34-b0af48044e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:18.012119 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:18.010450 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:18.012119 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:18.010581 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:18.012119 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:18.011071 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:18.012119 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:18.011176 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:18.200970 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:18.200922 2639 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="536190f352e88c88cbb3f2c0a2cb1d29dacbea1ced75b9e8ffcf8ba2887014b0" exitCode=0 Oct 31 21:15:18.201154 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:18.200996 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" event={"ID":"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a","Type":"ContainerDied","Data":"536190f352e88c88cbb3f2c0a2cb1d29dacbea1ced75b9e8ffcf8ba2887014b0"} Oct 31 21:15:19.010510 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:19.010257 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:19.010510 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:19.010423 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:19.215900 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:19.215296 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9kbsn" event={"ID":"ac6429fb-48a7-4691-9079-4ed168f1da32","Type":"ContainerStarted","Data":"6471b6203c89ca310b6ed3895816b07d29fc9c7bd41ee44a356587355256f100"} Oct 31 21:15:19.259484 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:19.259179 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9kbsn" podStartSLOduration=3.358326222 podStartE2EDuration="7.259165313s" totalImagesPullingTime="3.900839091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:19.258487937 +0000 UTC m=+7.720962750" watchObservedRunningTime="2026-10-31 21:15:19.259165313 +0000 UTC m=+7.721640126" Oct 31 21:15:20.014355 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:20.014260 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:20.014531 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:20.014410 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:20.014531 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:20.014421 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:20.014531 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:20.014495 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:20.223142 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:20.223098 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-42vxn" event={"ID":"672a2ce2-d45d-4f0c-a75a-b018105640aa","Type":"ContainerStarted","Data":"7c4327dd072976282f8af22df067445f40307eff2e96e757877e2c1cbdecae53"} Oct 31 21:15:20.285393 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:20.285315 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-42vxn" podStartSLOduration=3.298269772 podStartE2EDuration="8.285298066s" totalImagesPullingTime="4.987028294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:20.284936586 +0000 UTC m=+8.747411403" watchObservedRunningTime="2026-10-31 21:15:20.285298066 +0000 UTC m=+8.747772880" Oct 31 21:15:20.723168 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:20.723084 2639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:20.927364 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:20.927269 2639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:20.927872 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:20.927708 2639 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:21.009890 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:21.009775 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:21.010051 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:21.009892 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:21.230641 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:21.230607 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" event={"ID":"626d59af-b777-449a-87cc-23e763dd495f","Type":"ContainerStarted","Data":"0544ec2951fbf7a893c7a50f90caf5ed08e40e02cd23669772205cf437d71f36"} Oct 31 21:15:21.232044 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:21.232021 2639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:21.716635 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:21.716596 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:21.716822 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:21.716662 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:21.716822 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:21.716809 2639 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:21.716915 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:21.716874 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret podName:72751047-bc3c-4f2d-9655-e23d5211d346 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:29.716854827 +0000 UTC m=+18.179329635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret") pod "global-pull-secret-syncer-whzgx" (UID: "72751047-bc3c-4f2d-9655-e23d5211d346") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:21.717347 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:21.717313 2639 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:21.717424 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:21.717375 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:29.717362909 +0000 UTC m=+18.179837713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:21.818581 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:21.817900 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:21.818581 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:21.818085 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:21.818581 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:21.818107 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:21.818581 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:21.818120 2639 projected.go:196] Error preparing data for projected volume kube-api-access-fc8sb for pod openshift-network-diagnostics/network-check-target-tpl7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:21.818581 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:21.818183 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb podName:c929e38e-147c-41f9-ad34-b0af48044e09 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:29.818164073 +0000 UTC m=+18.280638887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fc8sb" (UniqueName: "kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb") pod "network-check-target-tpl7t" (UID: "c929e38e-147c-41f9-ad34-b0af48044e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:21.950378 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:21.950279 2639 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 21:15:22.011176 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:22.010957 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:22.011354 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:22.011213 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:22.011354 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:22.011262 2639 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T21:15:21.950303677Z","UUID":"72794ff0-9542-4847-a5b6-8fabce599986","Handler":null,"Name":"","Endpoint":""} Oct 31 21:15:22.011486 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:22.011464 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:22.011582 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:22.011559 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:22.014920 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:22.014728 2639 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 21:15:22.014920 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:22.014762 2639 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 21:15:22.235555 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:22.235521 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" event={"ID":"626d59af-b777-449a-87cc-23e763dd495f","Type":"ContainerStarted","Data":"6d6842a3d432aadb618f2e7605813f45dfba6a5490d2947c182d450e46216f62"} Oct 31 21:15:22.239511 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:22.239467 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxbcm" event={"ID":"47354b4e-2a35-40be-a539-07dad787bd97","Type":"ContainerStarted","Data":"e09e5ddff6a3a16351ebbcf00be29b251277b05ea2320a0c5ea9b78fbf06e166"} Oct 31 21:15:22.283787 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:22.283723 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jxbcm" podStartSLOduration=2.745090314 podStartE2EDuration="10.283708397s" totalImagesPullingTime="7.538618083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:22.283673746 +0000 UTC m=+10.746148573" watchObservedRunningTime="2026-10-31 21:15:22.283708397 +0000 UTC m=+10.746183210" Oct 31 21:15:23.009524 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:23.009488 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:23.009682 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:23.009652 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:23.243404 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:23.243356 2639 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="57aaa56b99c5215263015c045823720ad6277d38dd19c662874e51cbdc185d4f" exitCode=0 Oct 31 21:15:23.244025 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:23.243475 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" event={"ID":"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a","Type":"ContainerDied","Data":"57aaa56b99c5215263015c045823720ad6277d38dd19c662874e51cbdc185d4f"} Oct 31 21:15:23.246080 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:23.246045 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vx79w" event={"ID":"6805be97-0a2e-490c-a50c-6700f15c4abc","Type":"ContainerStarted","Data":"091c96f6da8c8f9cd9488056bac18da94b4d3d66a3f493e49abf48d112af93d8"} Oct 31 21:15:23.248837 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:23.248798 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" event={"ID":"626d59af-b777-449a-87cc-23e763dd495f","Type":"ContainerStarted","Data":"af3c59d702233434b0d8f9a17ac6fc492e974dc3a0d526b83178b863d0d662f3"} Oct 31 21:15:23.336261 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:23.336155 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" podStartSLOduration=3.186532756 podStartE2EDuration="11.336137712s" totalImagesPullingTime="8.149604956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:23.33613018 +0000 UTC m=+11.798604995" watchObservedRunningTime="2026-10-31 21:15:23.336137712 +0000 UTC m=+11.798612525" Oct 31 21:15:23.377100 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:23.377047 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vx79w" podStartSLOduration=3.643277602 podStartE2EDuration="11.377019028s" totalImagesPullingTime="7.733741426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:23.37652815 +0000 UTC m=+11.839002962" watchObservedRunningTime="2026-10-31 21:15:23.377019028 +0000 UTC m=+11.839493841" Oct 31 21:15:24.009793 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:24.009754 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:24.009996 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:24.009913 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:24.009996 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:24.009956 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:24.010108 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:24.010070 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:24.252575 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:24.252530 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x47n8" event={"ID":"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936","Type":"ContainerStarted","Data":"3f665b0464a5162cbd2e797ac17bd1ab5ea26c35c17af8577ba7cdbaa52a0848"} Oct 31 21:15:24.292868 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:24.292690 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x47n8" podStartSLOduration=4.744557687 podStartE2EDuration="12.292669102s" totalImagesPullingTime="7.548111415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:24.291801437 +0000 UTC m=+12.754276249" watchObservedRunningTime="2026-10-31 21:15:24.292669102 +0000 UTC m=+12.755143916" Oct 31 21:15:25.009872 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:25.009510 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:25.010035 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:25.009919 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:25.257589 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:25.257551 2639 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="191aa7f4f532eb81a807e9315ca8b35f2744c0f442338b9f859f4244dfd36cc1" exitCode=0 Oct 31 21:15:25.258272 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:25.257649 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" event={"ID":"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a","Type":"ContainerDied","Data":"191aa7f4f532eb81a807e9315ca8b35f2744c0f442338b9f859f4244dfd36cc1"} Oct 31 21:15:25.259905 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:25.259743 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrskg" event={"ID":"a4772ac2-3076-4099-89da-cb3be3d4387c","Type":"ContainerStarted","Data":"4ae4de403fe33c946cd6843501cb69d50979676f9e1512db7e176590bac92a7c"} Oct 31 21:15:26.013907 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:26.013876 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:26.013907 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:26.013905 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:26.014146 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:26.014012 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:26.014146 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:26.014127 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:26.265766 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:26.265677 2639 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="c37852e3129d121a35e61aafe9441bedbc60146a0e305682948435acb040601f" exitCode=0 Oct 31 21:15:26.266162 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:26.265753 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" event={"ID":"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a","Type":"ContainerDied","Data":"c37852e3129d121a35e61aafe9441bedbc60146a0e305682948435acb040601f"} Oct 31 21:15:26.324238 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:26.324183 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mrskg" podStartSLOduration=4.064506124 podStartE2EDuration="14.324166353s" totalImagesPullingTime="10.259660229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:25.373411112 +0000 UTC m=+13.835885927" watchObservedRunningTime="2026-10-31 21:15:26.324166353 +0000 UTC m=+14.786641206" Oct 31 21:15:27.009716 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:27.009683 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:27.009954 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:27.009864 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:28.009546 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:28.009457 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:28.009976 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:28.009607 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:28.009976 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:28.009911 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:28.010096 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:28.009987 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:28.273059 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:28.272965 2639 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="8a26ab298fcf190d8b2a57c31530f3b6236825d2472274f3e7dc0f98f09a3b85" exitCode=0 Oct 31 21:15:28.273059 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:28.273031 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" event={"ID":"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a","Type":"ContainerDied","Data":"8a26ab298fcf190d8b2a57c31530f3b6236825d2472274f3e7dc0f98f09a3b85"} Oct 31 21:15:29.009883 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.009865 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:29.010202 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:29.009965 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:29.278792 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.278388 2639 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="b6692948d28a2cf0b45ae0d23a05ba8035a9f89ec279aa95ce55bee2db96cab2" exitCode=0 Oct 31 21:15:29.278792 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.278469 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" event={"ID":"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a","Type":"ContainerDied","Data":"b6692948d28a2cf0b45ae0d23a05ba8035a9f89ec279aa95ce55bee2db96cab2"} Oct 31 21:15:29.282102 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.282077 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" event={"ID":"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6","Type":"ContainerStarted","Data":"3f8831fc32b88bc52f8503f25f2a6c79a5a08c1461f690558e551db04b6b821e"} Oct 31 21:15:29.282203 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.282145 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" event={"ID":"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6","Type":"ContainerStarted","Data":"e497b1cdd9f8fb9f2e5e165ae5853b18f8f5a10e07fccfbc05d1eb68f82f684e"} Oct 31 21:15:29.282203 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.282175 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" event={"ID":"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6","Type":"ContainerStarted","Data":"d80060bf67ddf161c2060098b0cee43750bc86492e2ba195c4c30e5584511822"} Oct 31 21:15:29.282261 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.282204 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" event={"ID":"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6","Type":"ContainerStarted","Data":"c42d1a295ea4a13358b2ee64947d451f6f2b990a8a4873775829712a88487ed2"} Oct 31 21:15:29.782453 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.782417 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:29.782609 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.782464 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:29.782609 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:29.782552 2639 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:29.782609 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:29.782583 2639 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:29.782609 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:29.782610 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret podName:72751047-bc3c-4f2d-9655-e23d5211d346 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.782596187 +0000 UTC m=+34.245070991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret") pod "global-pull-secret-syncer-whzgx" (UID: "72751047-bc3c-4f2d-9655-e23d5211d346") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:29.782756 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:29.782639 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.782626129 +0000 UTC m=+34.245100920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:29.883790 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:29.883752 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:29.883936 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:29.883886 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:29.883936 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:29.883900 2639 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:29.883936 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:29.883910 2639 projected.go:196] Error preparing data for projected volume kube-api-access-fc8sb for pod openshift-network-diagnostics/network-check-target-tpl7t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:29.884043 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:29.883959 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb podName:c929e38e-147c-41f9-ad34-b0af48044e09 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.883946729 +0000 UTC m=+34.346421533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fc8sb" (UniqueName: "kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb") pod "network-check-target-tpl7t" (UID: "c929e38e-147c-41f9-ad34-b0af48044e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:30.013374 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:30.013346 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:30.013908 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:30.013379 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:30.013908 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:30.013448 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:30.013908 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:30.013555 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:30.287488 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:30.287452 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" event={"ID":"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6","Type":"ContainerStarted","Data":"b634eaab5c79e6d6b664aa8409e3d12540452cafadeb7d04241ab242ccdd89c8"} Oct 31 21:15:30.287661 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:30.287535 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" event={"ID":"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6","Type":"ContainerStarted","Data":"8505662bfcc9c6165fecb74fc6334830dc7d847a216888a4be9d4141d2182926"} Oct 31 21:15:30.290785 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:30.290746 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" event={"ID":"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a","Type":"ContainerStarted","Data":"1a13380605c274fd768033b4c3349bb3a1b17032cca7c1b342ae51f432f88b4d"} Oct 31 21:15:31.010431 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:31.010391 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:31.010613 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:31.010508 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:32.010906 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:32.010869 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:32.011428 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:32.010972 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:32.011428 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:32.011055 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:32.011648 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:32.011167 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:32.298657 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:32.298567 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" event={"ID":"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6","Type":"ContainerStarted","Data":"2e15b5ef3be4c78359e5e0f3d89f5860b055b5cfbd7709fec4ed32ef5be03ce4"} Oct 31 21:15:33.010030 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:33.009977 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:33.010247 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:33.010122 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:34.010300 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.010266 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:34.010911 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.010303 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:34.010911 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:34.010422 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:34.010911 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:34.010549 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:34.306972 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.306873 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" event={"ID":"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6","Type":"ContainerStarted","Data":"30e165558fe1aef99b3b0859a59dd87905bcaeccdde41439f3325e9339036b14"} Oct 31 21:15:34.307257 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.307225 2639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:34.307406 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.307308 2639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:34.307406 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.307378 2639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:34.323910 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.323882 2639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:34.327902 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.327881 2639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:34.398179 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.398127 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" podStartSLOduration=7.734685914 podStartE2EDuration="22.39811305s" totalImagesPullingTime="14.663427136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:34.397811772 +0000 UTC m=+22.860286585" watchObservedRunningTime="2026-10-31 21:15:34.39811305 +0000 UTC m=+22.860587882" Oct 31 21:15:34.398427 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:34.398405 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2l8b7" podStartSLOduration=11.782583164 podStartE2EDuration="22.398400272s" totalImagesPullingTime="10.615817108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:30.361111849 +0000 UTC m=+18.823586663" watchObservedRunningTime="2026-10-31 21:15:34.398400272 +0000 UTC m=+22.860875084" Oct 31 21:15:35.010339 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:35.010284 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:35.010757 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:35.010413 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:35.651718 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:35.651684 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-whzgx"] Oct 31 21:15:35.652001 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:35.651808 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:35.652001 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:35.651891 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:35.654952 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:35.654923 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tpl7t"] Oct 31 21:15:35.655088 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:35.655056 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:35.655199 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:35.655173 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:35.663894 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:35.663861 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zc54c"] Oct 31 21:15:35.664037 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:35.663986 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:35.664097 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:35.664079 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:37.009490 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:37.009446 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:37.009922 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:37.009452 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:37.009922 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:37.009561 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:37.009922 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:37.009674 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:38.010532 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:38.010436 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:38.010921 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:38.010610 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:39.009565 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:39.009528 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:39.009565 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:39.009552 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:39.009801 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:39.009644 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:39.009801 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:39.009752 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:40.010221 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:40.010185 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:40.010869 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:40.010307 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-whzgx" podUID="72751047-bc3c-4f2d-9655-e23d5211d346" Oct 31 21:15:41.010422 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.010386 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:41.010811 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.010385 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:41.010811 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:41.010502 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zc54c" podUID="3616b3de-b6e6-497f-8546-8fef766c309f" Oct 31 21:15:41.010811 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:41.010569 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tpl7t" podUID="c929e38e-147c-41f9-ad34-b0af48044e09" Oct 31 21:15:41.818013 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.817824 2639 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeReady" Oct 31 21:15:41.818176 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.818120 2639 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 21:15:41.865343 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.865297 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cps44"] Oct 31 21:15:41.868887 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.868842 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w96gb"] Oct 31 21:15:41.869045 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.868992 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cps44" Oct 31 21:15:41.871977 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.871950 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7fh65"] Oct 31 21:15:41.872169 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.872150 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w96gb" Oct 31 21:15:41.874662 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.874642 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-574b9c7469-m4qfg"] Oct 31 21:15:41.875282 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.875235 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7fh65" Oct 31 21:15:41.880727 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.880693 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bnbr9"] Oct 31 21:15:41.880893 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.880867 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:41.884224 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.884202 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-68d4c65fcc-47wdn"] Oct 31 21:15:41.884363 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.884346 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:41.887248 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.887219 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.887410 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.887383 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql"] Oct 31 21:15:41.888214 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.888192 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:41.889613 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.889592 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.890360 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.890345 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6"] Oct 31 21:15:41.890423 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.890403 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" Oct 31 21:15:41.893106 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.893085 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm"] Oct 31 21:15:41.893233 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.893207 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:41.896097 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.896080 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk"] Oct 31 21:15:41.896228 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.896213 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:41.899091 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.899062 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sh7vx"] Oct 31 21:15:41.899203 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.899107 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:41.902501 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.902473 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cg8tz"] Oct 31 21:15:41.902619 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.902583 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:41.904843 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.904820 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 21:15:41.905816 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.905795 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c58487768-swgvs"] Oct 31 21:15:41.906023 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.905920 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:41.908717 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.908695 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-28gfg\"" Oct 31 21:15:41.909852 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.909830 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr"] Oct 31 21:15:41.909963 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.909944 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:41.912237 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.912217 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 21:15:41.912835 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.912812 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j"] Oct 31 21:15:41.912952 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.912928 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:41.916535 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.916507 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wj49j"] Oct 31 21:15:41.916640 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.916587 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" Oct 31 21:15:41.919921 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.919890 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mb4ql"] Oct 31 21:15:41.920084 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.920063 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:41.923079 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.923033 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cps44"] Oct 31 21:15:41.923220 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.923099 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w96gb"] Oct 31 21:15:41.923220 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.923123 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7fh65"] Oct 31 21:15:41.923220 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.923179 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:41.924544 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.924492 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql"] Oct 31 21:15:41.925647 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.925626 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6"] Oct 31 21:15:41.927221 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.927197 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr"] Oct 31 21:15:41.927516 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.927496 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 21:15:41.945742 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.945340 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cg8tz"] Oct 31 21:15:41.945742 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.945455 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bnbr9"] Oct 31 21:15:41.945949 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.945922 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm"] Oct 31 21:15:41.947585 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.947560 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wj49j"] Oct 31 21:15:41.949743 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.948340 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j"] Oct 31 21:15:41.949743 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.949005 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68d4c65fcc-47wdn"] Oct 31 21:15:41.950033 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.949835 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sh7vx"] Oct 31 21:15:41.956191 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.956022 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-574b9c7469-m4qfg"] Oct 31 21:15:41.959842 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.959809 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mb4ql"] Oct 31 21:15:41.961827 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.961802 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk"] Oct 31 21:15:41.963427 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963400 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:41.963599 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963439 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927v6\" (UniqueName: \"kubernetes.io/projected/f7d03911-7706-47b8-928e-6ffea21fdb26-kube-api-access-927v6\") pod \"volume-data-source-validator-58c9f5f46d-swfql\" (UID: \"f7d03911-7706-47b8-928e-6ffea21fdb26\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" Oct 31 21:15:41.963599 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963467 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:41.963599 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963515 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktw9c\" (UniqueName: \"kubernetes.io/projected/4bf7148c-69be-45a9-bd56-7457a3929df8-kube-api-access-ktw9c\") pod \"kas-connection-checker-685895c7c9-7fh65\" (UID: \"4bf7148c-69be-45a9-bd56-7457a3929df8\") " pod="kube-system/kas-connection-checker-685895c7c9-7fh65" Oct 31 21:15:41.963599 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963585 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:41.963780 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963624 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxpcl\" (UniqueName: \"kubernetes.io/projected/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-kube-api-access-bxpcl\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:41.963780 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963649 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-tmp\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:41.963780 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963675 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:41.963780 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963701 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:41.963780 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963722 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b23775a-33a5-47b0-ac74-fb3862912542-serving-cert\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:41.963780 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963743 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-trusted-ca\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:41.963780 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963770 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:41.964073 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963792 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:41.964073 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963818 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/31682289-5cba-480d-88a4-c71e9cd6244b-kube-api-access-lxlcr\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:41.964073 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963883 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:41.964073 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963940 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qmg\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:41.964073 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.963980 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:41.964073 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964000 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a47228-9b40-4833-bb84-970f4782e67b-serving-cert\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:41.964073 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964047 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hljm\" (UniqueName: \"kubernetes.io/projected/1b23775a-33a5-47b0-ac74-fb3862912542-kube-api-access-7hljm\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964075 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b445d\" (UniqueName: \"kubernetes.io/projected/d2a47228-9b40-4833-bb84-970f4782e67b-kube-api-access-b445d\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964106 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964124 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964148 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964166 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964204 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5kk\" (UniqueName: \"kubernetes.io/projected/9f8d2547-a263-47ff-aaa9-780ea67176a5-kube-api-access-gs5kk\") pod \"kas-connection-checker-685895c7c9-w96gb\" (UID: \"9f8d2547-a263-47ff-aaa9-780ea67176a5\") " pod="kube-system/kas-connection-checker-685895c7c9-w96gb" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964248 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98zhq\" (UniqueName: \"kubernetes.io/projected/7ed572b5-9b0c-4924-af2f-87703ed1b5e6-kube-api-access-98zhq\") pod \"kas-connection-checker-685895c7c9-cps44\" (UID: \"7ed572b5-9b0c-4924-af2f-87703ed1b5e6\") " pod="kube-system/kas-connection-checker-685895c7c9-cps44" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964299 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964316 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rxlh\" (UniqueName: \"kubernetes.io/projected/e49a2d50-dc82-4983-9425-053dfe903ddd-kube-api-access-4rxlh\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:41.964439 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964391 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31682289-5cba-480d-88a4-c71e9cd6244b-config\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964461 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw9ch\" (UniqueName: \"kubernetes.io/projected/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-kube-api-access-cw9ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964501 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964525 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31682289-5cba-480d-88a4-c71e9cd6244b-serving-cert\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964558 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-default-certificate\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964608 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-config\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964634 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964665 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fwg\" (UniqueName: \"kubernetes.io/projected/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-kube-api-access-t7fwg\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964708 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vtd\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964748 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964773 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964799 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964823 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964849 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:41.964884 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964871 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-snapshots\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:41.965536 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964901 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-stats-auth\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:41.965536 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964924 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:41.965536 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964971 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:41.965536 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.964997 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:41.965536 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.965021 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:41.965536 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.965046 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:41.965890 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.965871 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c58487768-swgvs"] Oct 31 21:15:41.987599 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.987577 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.988650 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988629 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 21:15:41.988770 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988639 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.988770 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988662 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.988770 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988702 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 21:15:41.988770 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988725 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.988940 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988839 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.988940 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988851 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 21:15:41.988940 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988900 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.988940 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988926 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.989120 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.988975 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 21:15:41.989120 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989024 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 21:15:41.989120 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989067 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.989120 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989071 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 21:15:41.989120 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989107 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 21:15:41.989304 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989126 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.989381 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989352 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.989381 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989365 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 21:15:41.989478 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989398 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.989478 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989406 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.989478 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989424 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.989478 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989441 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.989478 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989403 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 21:15:41.989478 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989472 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.989719 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989473 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.989719 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989621 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.989719 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989620 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:41.989719 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:41.989645 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:42.002348 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.002301 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 21:15:42.009661 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.009637 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:42.014284 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014259 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014310 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014408 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014491 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014546 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014551 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014572 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014586 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014605 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014630 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014642 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qc9xz\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014653 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014687 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cs6xw\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014696 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gpdvx\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014634 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014698 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-48hbl\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014703 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 21:15:42.014723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014725 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5xwjc\"" Oct 31 21:15:42.015362 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014755 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nt72v\"" Oct 31 21:15:42.015362 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014758 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nvc4l\"" Oct 31 21:15:42.015362 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014814 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dwlts\"" Oct 31 21:15:42.015362 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014844 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 21:15:42.015362 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014853 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7pbqt\"" Oct 31 21:15:42.015362 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014918 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m2xrj\"" Oct 31 21:15:42.015362 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.014961 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 21:15:42.015362 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.015029 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-nhmjw\"" Oct 31 21:15:42.015362 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.015073 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nq8jq\"" Oct 31 21:15:42.024369 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.024349 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 21:15:42.024486 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.024419 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j68t2\"" Oct 31 21:15:42.065443 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065410 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxpcl\" (UniqueName: \"kubernetes.io/projected/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-kube-api-access-bxpcl\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.065443 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065443 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-tmp\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.065683 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065463 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:42.065683 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065482 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:42.065683 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065499 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b23775a-33a5-47b0-ac74-fb3862912542-serving-cert\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:42.065683 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065522 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-trusted-ca\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:42.065683 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.065590 2639 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:42.065683 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065620 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:42.065683 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.065655 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls podName:c01f3957-4a49-41ba-812b-ea2e74d3a7a8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:42.565635364 +0000 UTC m=+31.028110175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8r9vm" (UID: "c01f3957-4a49-41ba-812b-ea2e74d3a7a8") : secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065689 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065716 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/31682289-5cba-480d-88a4-c71e9cd6244b-kube-api-access-lxlcr\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065744 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065775 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qmg\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065801 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065824 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a47228-9b40-4833-bb84-970f4782e67b-serving-cert\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065862 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hljm\" (UniqueName: \"kubernetes.io/projected/1b23775a-33a5-47b0-ac74-fb3862912542-kube-api-access-7hljm\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065887 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b445d\" (UniqueName: \"kubernetes.io/projected/d2a47228-9b40-4833-bb84-970f4782e67b-kube-api-access-b445d\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065913 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065921 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-tmp\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065938 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.065966 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.066022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066008 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066043 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5kk\" (UniqueName: \"kubernetes.io/projected/9f8d2547-a263-47ff-aaa9-780ea67176a5-kube-api-access-gs5kk\") pod \"kas-connection-checker-685895c7c9-w96gb\" (UID: \"9f8d2547-a263-47ff-aaa9-780ea67176a5\") " pod="kube-system/kas-connection-checker-685895c7c9-w96gb" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066079 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98zhq\" (UniqueName: \"kubernetes.io/projected/7ed572b5-9b0c-4924-af2f-87703ed1b5e6-kube-api-access-98zhq\") pod \"kas-connection-checker-685895c7c9-cps44\" (UID: \"7ed572b5-9b0c-4924-af2f-87703ed1b5e6\") " pod="kube-system/kas-connection-checker-685895c7c9-cps44" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066124 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.066143 2639 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066149 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxlh\" (UniqueName: \"kubernetes.io/projected/e49a2d50-dc82-4983-9425-053dfe903ddd-kube-api-access-4rxlh\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066175 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31682289-5cba-480d-88a4-c71e9cd6244b-config\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.066194 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls podName:e49a2d50-dc82-4983-9425-053dfe903ddd nodeName:}" failed. No retries permitted until 2026-10-31 21:15:42.566179413 +0000 UTC m=+31.028654219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-k4qxr" (UID: "e49a2d50-dc82-4983-9425-053dfe903ddd") : secret "samples-operator-tls" not found Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066234 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw9ch\" (UniqueName: \"kubernetes.io/projected/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-kube-api-access-cw9ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066261 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066276 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.066373 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.066386 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b9c7469-m4qfg: secret "image-registry-tls" not found Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066406 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.066428 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls podName:ce941b3a-da84-4b86-944c-89f9f464a27f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:42.566414736 +0000 UTC m=+31.028889555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls") pod "image-registry-574b9c7469-m4qfg" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f") : secret "image-registry-tls" not found Oct 31 21:15:42.066648 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066484 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066843 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066283 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31682289-5cba-480d-88a4-c71e9cd6244b-serving-cert\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066870 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31682289-5cba-480d-88a4-c71e9cd6244b-config\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066902 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-default-certificate\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066927 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-config\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066954 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjqft\" (UniqueName: \"kubernetes.io/projected/e6924bea-83c3-443f-a95d-581edf9aaf7b-kube-api-access-mjqft\") pod \"network-check-source-6f78d59f99-cbc4j\" (UID: \"e6924bea-83c3-443f-a95d-581edf9aaf7b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.066990 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067015 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fwg\" (UniqueName: \"kubernetes.io/projected/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-kube-api-access-t7fwg\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067036 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067064 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vtd\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067085 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcnhc\" (UniqueName: \"kubernetes.io/projected/b6f98741-1d48-4124-9acb-8c0bf31dbe60-kube-api-access-pcnhc\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067112 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/add13542-315e-4dde-a5fa-728aaf6fdc6c-cert\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067132 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwtqf\" (UniqueName: \"kubernetes.io/projected/add13542-315e-4dde-a5fa-728aaf6fdc6c-kube-api-access-mwtqf\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067155 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067178 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.067371 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067200 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067220 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067232 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-trusted-ca\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067241 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067263 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-snapshots\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067290 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-stats-auth\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067315 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067356 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f98741-1d48-4124-9acb-8c0bf31dbe60-tmp-dir\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067397 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067420 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f98741-1d48-4124-9acb-8c0bf31dbe60-config-volume\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067443 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067468 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067497 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067525 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067547 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927v6\" (UniqueName: \"kubernetes.io/projected/f7d03911-7706-47b8-928e-6ffea21fdb26-kube-api-access-927v6\") pod \"volume-data-source-validator-58c9f5f46d-swfql\" (UID: \"f7d03911-7706-47b8-928e-6ffea21fdb26\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067550 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.068116 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067570 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.068902 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067593 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktw9c\" (UniqueName: \"kubernetes.io/projected/4bf7148c-69be-45a9-bd56-7457a3929df8-kube-api-access-ktw9c\") pod \"kas-connection-checker-685895c7c9-7fh65\" (UID: \"4bf7148c-69be-45a9-bd56-7457a3929df8\") " pod="kube-system/kas-connection-checker-685895c7c9-7fh65" Oct 31 21:15:42.068902 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.067619 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.068902 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.067760 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:42.567744675 +0000 UTC m=+31.030219489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : configmap references non-existent config key: service-ca.crt Oct 31 21:15:42.068902 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.068425 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.068902 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.068439 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:42.068902 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.068533 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:42.068902 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.068717 2639 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 21:15:42.068902 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.068786 2639 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 21:15:42.068902 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.068845 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert podName:0bd131db-e6d0-46c6-9a4b-85d3b541a9ec nodeName:}" failed. No retries permitted until 2026-10-31 21:15:42.568827999 +0000 UTC m=+31.031302807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sh7vx" (UID: "0bd131db-e6d0-46c6-9a4b-85d3b541a9ec") : secret "networking-console-plugin-cert" not found Oct 31 21:15:42.070950 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.070919 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.071056 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.070997 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b23775a-33a5-47b0-ac74-fb3862912542-serving-cert\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:42.071056 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.071023 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a47228-9b40-4833-bb84-970f4782e67b-serving-cert\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.071148 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.071069 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.071148 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.071079 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.071252 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.071165 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:42.571147564 +0000 UTC m=+31.033622362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : secret "router-metrics-certs-default" not found Oct 31 21:15:42.071425 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.071404 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:42.071425 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.071422 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c58487768-swgvs: secret "image-registry-tls" not found Oct 31 21:15:42.071584 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.071459 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-default-certificate\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.071584 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.071530 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.071709 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.071590 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls podName:5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b nodeName:}" failed. No retries permitted until 2026-10-31 21:15:42.571576514 +0000 UTC m=+31.034051305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls") pod "image-registry-c58487768-swgvs" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b") : secret "image-registry-tls" not found Oct 31 21:15:42.071709 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.071622 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31682289-5cba-480d-88a4-c71e9cd6244b-serving-cert\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:42.071808 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.071752 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.072274 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.071935 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.072274 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.072198 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-config\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:42.072274 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.072213 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.072274 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.072232 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-snapshots\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.072556 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.072392 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.073584 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.073566 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:42.073769 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.073750 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-stats-auth\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.091314 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.091290 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/31682289-5cba-480d-88a4-c71e9cd6244b-kube-api-access-lxlcr\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:42.091596 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.091573 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxpcl\" (UniqueName: \"kubernetes.io/projected/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-kube-api-access-bxpcl\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.091682 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.091602 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qmg\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.093012 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.092990 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxlh\" (UniqueName: \"kubernetes.io/projected/e49a2d50-dc82-4983-9425-053dfe903ddd-kube-api-access-4rxlh\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:42.094279 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.094259 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5kk\" (UniqueName: \"kubernetes.io/projected/9f8d2547-a263-47ff-aaa9-780ea67176a5-kube-api-access-gs5kk\") pod \"kas-connection-checker-685895c7c9-w96gb\" (UID: \"9f8d2547-a263-47ff-aaa9-780ea67176a5\") " pod="kube-system/kas-connection-checker-685895c7c9-w96gb" Oct 31 21:15:42.094468 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.094447 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98zhq\" (UniqueName: \"kubernetes.io/projected/7ed572b5-9b0c-4924-af2f-87703ed1b5e6-kube-api-access-98zhq\") pod \"kas-connection-checker-685895c7c9-cps44\" (UID: \"7ed572b5-9b0c-4924-af2f-87703ed1b5e6\") " pod="kube-system/kas-connection-checker-685895c7c9-cps44" Oct 31 21:15:42.096098 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.096073 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fwg\" (UniqueName: \"kubernetes.io/projected/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-kube-api-access-t7fwg\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:42.096237 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.096217 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktw9c\" (UniqueName: \"kubernetes.io/projected/4bf7148c-69be-45a9-bd56-7457a3929df8-kube-api-access-ktw9c\") pod \"kas-connection-checker-685895c7c9-7fh65\" (UID: \"4bf7148c-69be-45a9-bd56-7457a3929df8\") " pod="kube-system/kas-connection-checker-685895c7c9-7fh65" Oct 31 21:15:42.099022 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.098999 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.099391 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.099366 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hljm\" (UniqueName: \"kubernetes.io/projected/1b23775a-33a5-47b0-ac74-fb3862912542-kube-api-access-7hljm\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:42.099633 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.099610 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.099761 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.099741 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vtd\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.100156 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.100131 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw9ch\" (UniqueName: \"kubernetes.io/projected/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-kube-api-access-cw9ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:42.100236 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.100219 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b445d\" (UniqueName: \"kubernetes.io/projected/d2a47228-9b40-4833-bb84-970f4782e67b-kube-api-access-b445d\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.100533 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.100515 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-927v6\" (UniqueName: \"kubernetes.io/projected/f7d03911-7706-47b8-928e-6ffea21fdb26-kube-api-access-927v6\") pod \"volume-data-source-validator-58c9f5f46d-swfql\" (UID: \"f7d03911-7706-47b8-928e-6ffea21fdb26\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" Oct 31 21:15:42.169019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.168939 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjqft\" (UniqueName: \"kubernetes.io/projected/e6924bea-83c3-443f-a95d-581edf9aaf7b-kube-api-access-mjqft\") pod \"network-check-source-6f78d59f99-cbc4j\" (UID: \"e6924bea-83c3-443f-a95d-581edf9aaf7b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" Oct 31 21:15:42.169019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.168979 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.169019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.169001 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcnhc\" (UniqueName: \"kubernetes.io/projected/b6f98741-1d48-4124-9acb-8c0bf31dbe60-kube-api-access-pcnhc\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.169019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.169019 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/add13542-315e-4dde-a5fa-728aaf6fdc6c-cert\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:42.169370 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.169034 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwtqf\" (UniqueName: \"kubernetes.io/projected/add13542-315e-4dde-a5fa-728aaf6fdc6c-kube-api-access-mwtqf\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:42.169370 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.169097 2639 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 21:15:42.169370 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.169162 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls podName:b6f98741-1d48-4124-9acb-8c0bf31dbe60 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:42.669143118 +0000 UTC m=+31.131617928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls") pod "dns-default-wj49j" (UID: "b6f98741-1d48-4124-9acb-8c0bf31dbe60") : secret "dns-default-metrics-tls" not found Oct 31 21:15:42.169370 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.169208 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f98741-1d48-4124-9acb-8c0bf31dbe60-tmp-dir\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.169370 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.169247 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f98741-1d48-4124-9acb-8c0bf31dbe60-config-volume\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.169598 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.169582 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f98741-1d48-4124-9acb-8c0bf31dbe60-tmp-dir\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.169777 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.169760 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f98741-1d48-4124-9acb-8c0bf31dbe60-config-volume\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.171573 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.171550 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/add13542-315e-4dde-a5fa-728aaf6fdc6c-cert\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:42.186009 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.185973 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cps44" Oct 31 21:15:42.192895 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.192870 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcnhc\" (UniqueName: \"kubernetes.io/projected/b6f98741-1d48-4124-9acb-8c0bf31dbe60-kube-api-access-pcnhc\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.193234 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.193216 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjqft\" (UniqueName: \"kubernetes.io/projected/e6924bea-83c3-443f-a95d-581edf9aaf7b-kube-api-access-mjqft\") pod \"network-check-source-6f78d59f99-cbc4j\" (UID: \"e6924bea-83c3-443f-a95d-581edf9aaf7b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" Oct 31 21:15:42.194033 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.194010 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwtqf\" (UniqueName: \"kubernetes.io/projected/add13542-315e-4dde-a5fa-728aaf6fdc6c-kube-api-access-mwtqf\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:42.194957 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.194942 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w96gb" Oct 31 21:15:42.203642 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.203588 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7fh65" Oct 31 21:15:42.218062 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.218033 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:42.248634 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.248605 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" Oct 31 21:15:42.257175 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.256935 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:42.278114 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.277703 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:42.290727 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.290483 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:42.310180 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.309966 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" Oct 31 21:15:42.328446 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.328418 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:42.510401 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.510377 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w96gb"] Oct 31 21:15:42.514306 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:42.514255 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8d2547_a263_47ff_aaa9_780ea67176a5.slice/crio-946256b8d18c3f7253e86f5402267c7e80521b1a1326694aab8d6727c85c52fd WatchSource:0}: Error finding container 946256b8d18c3f7253e86f5402267c7e80521b1a1326694aab8d6727c85c52fd: Status 404 returned error can't find the container with id 946256b8d18c3f7253e86f5402267c7e80521b1a1326694aab8d6727c85c52fd Oct 31 21:15:42.514606 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.514584 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cps44"] Oct 31 21:15:42.516764 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:42.516726 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed572b5_9b0c_4924_af2f_87703ed1b5e6.slice/crio-297bc8e6371cfa9f4f146cee5f2ec511763b3867bdd0ce42e7a9d610c7f5889d WatchSource:0}: Error finding container 297bc8e6371cfa9f4f146cee5f2ec511763b3867bdd0ce42e7a9d610c7f5889d: Status 404 returned error can't find the container with id 297bc8e6371cfa9f4f146cee5f2ec511763b3867bdd0ce42e7a9d610c7f5889d Oct 31 21:15:42.521172 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.521145 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7fh65"] Oct 31 21:15:42.528500 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:42.528451 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf7148c_69be_45a9_bd56_7457a3929df8.slice/crio-2dd955e49284c880e75f26845637a762eb0802737a3ea7b18174e9f566869edf WatchSource:0}: Error finding container 2dd955e49284c880e75f26845637a762eb0802737a3ea7b18174e9f566869edf: Status 404 returned error can't find the container with id 2dd955e49284c880e75f26845637a762eb0802737a3ea7b18174e9f566869edf Oct 31 21:15:42.547576 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.547546 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-bnbr9"] Oct 31 21:15:42.552850 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:42.552816 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a47228_9b40_4833_bb84_970f4782e67b.slice/crio-fd425a165bedcbef0d9b6e7c11ab02987a2dc413dede5b11e287dd50ce339fff WatchSource:0}: Error finding container fd425a165bedcbef0d9b6e7c11ab02987a2dc413dede5b11e287dd50ce339fff: Status 404 returned error can't find the container with id fd425a165bedcbef0d9b6e7c11ab02987a2dc413dede5b11e287dd50ce339fff Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.573005 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.573090 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.573130 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.573173 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.573211 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.573275 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.573304 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573464 2639 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573526 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:43.573507664 +0000 UTC m=+32.035982473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : secret "router-metrics-certs-default" not found Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573588 2639 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573621 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert podName:0bd131db-e6d0-46c6-9a4b-85d3b541a9ec nodeName:}" failed. No retries permitted until 2026-10-31 21:15:43.573610637 +0000 UTC m=+32.036085445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sh7vx" (UID: "0bd131db-e6d0-46c6-9a4b-85d3b541a9ec") : secret "networking-console-plugin-cert" not found Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573682 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:43.573673162 +0000 UTC m=+32.036147971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : configmap references non-existent config key: service-ca.crt Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573736 2639 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573765 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls podName:c01f3957-4a49-41ba-812b-ea2e74d3a7a8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:43.57375587 +0000 UTC m=+32.036230680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8r9vm" (UID: "c01f3957-4a49-41ba-812b-ea2e74d3a7a8") : secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:42.574049 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573819 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:42.575005 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573829 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b9c7469-m4qfg: secret "image-registry-tls" not found Oct 31 21:15:42.575005 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573857 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls podName:ce941b3a-da84-4b86-944c-89f9f464a27f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:43.573848059 +0000 UTC m=+32.036322853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls") pod "image-registry-574b9c7469-m4qfg" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f") : secret "image-registry-tls" not found Oct 31 21:15:42.575005 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573911 2639 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 21:15:42.575005 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573938 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls podName:e49a2d50-dc82-4983-9425-053dfe903ddd nodeName:}" failed. No retries permitted until 2026-10-31 21:15:43.573930145 +0000 UTC m=+32.036404952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-k4qxr" (UID: "e49a2d50-dc82-4983-9425-053dfe903ddd") : secret "samples-operator-tls" not found Oct 31 21:15:42.575005 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573988 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:42.575005 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.573997 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c58487768-swgvs: secret "image-registry-tls" not found Oct 31 21:15:42.575005 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.574024 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls podName:5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b nodeName:}" failed. No retries permitted until 2026-10-31 21:15:43.574014894 +0000 UTC m=+32.036489699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls") pod "image-registry-c58487768-swgvs" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b") : secret "image-registry-tls" not found Oct 31 21:15:42.619061 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.618982 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql"] Oct 31 21:15:42.622347 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:42.622305 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d03911_7706_47b8_928e_6ffea21fdb26.slice/crio-16a63ca010aea713e9ad65816a00f6398257af06dac75fed21210be7279314b5 WatchSource:0}: Error finding container 16a63ca010aea713e9ad65816a00f6398257af06dac75fed21210be7279314b5: Status 404 returned error can't find the container with id 16a63ca010aea713e9ad65816a00f6398257af06dac75fed21210be7279314b5 Oct 31 21:15:42.662733 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.662698 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6"] Oct 31 21:15:42.668244 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:42.668209 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a28634_04b5_4d1b_bf14_6bdc3b894f2d.slice/crio-b46c20bb76e6bce8e4e792dc5ccb61793ff1e4d945aeda835f6f484a8123866e WatchSource:0}: Error finding container b46c20bb76e6bce8e4e792dc5ccb61793ff1e4d945aeda835f6f484a8123866e: Status 404 returned error can't find the container with id b46c20bb76e6bce8e4e792dc5ccb61793ff1e4d945aeda835f6f484a8123866e Oct 31 21:15:42.675822 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.675563 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:42.675822 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.675693 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cg8tz"] Oct 31 21:15:42.675822 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.675736 2639 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 21:15:42.675822 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:42.675796 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls podName:b6f98741-1d48-4124-9acb-8c0bf31dbe60 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:43.675774911 +0000 UTC m=+32.138249720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls") pod "dns-default-wj49j" (UID: "b6f98741-1d48-4124-9acb-8c0bf31dbe60") : secret "dns-default-metrics-tls" not found Oct 31 21:15:42.678857 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:42.678832 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b23775a_33a5_47b0_ac74_fb3862912542.slice/crio-23342e2a936832b22f8ab613a00a5f38fff3fa37c4299db5160ef977d669bb0c WatchSource:0}: Error finding container 23342e2a936832b22f8ab613a00a5f38fff3fa37c4299db5160ef977d669bb0c: Status 404 returned error can't find the container with id 23342e2a936832b22f8ab613a00a5f38fff3fa37c4299db5160ef977d669bb0c Oct 31 21:15:42.685108 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.685080 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk"] Oct 31 21:15:42.685182 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.685150 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mb4ql"] Oct 31 21:15:42.689703 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:42.689629 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j"] Oct 31 21:15:42.695086 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:42.695060 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6924bea_83c3_443f_a95d_581edf9aaf7b.slice/crio-037b736e04958f93ebd6345cab2da0d2b65d0a0d939e5c7283a74e18467e7c3a WatchSource:0}: Error finding container 037b736e04958f93ebd6345cab2da0d2b65d0a0d939e5c7283a74e18467e7c3a: Status 404 returned error can't find the container with id 037b736e04958f93ebd6345cab2da0d2b65d0a0d939e5c7283a74e18467e7c3a Oct 31 21:15:43.010267 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.010187 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:43.010267 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.010226 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:43.025885 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.025854 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflcb\"" Oct 31 21:15:43.026277 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.025948 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvv8l\"" Oct 31 21:15:43.026277 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.026041 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 21:15:43.334191 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.334107 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" event={"ID":"e6924bea-83c3-443f-a95d-581edf9aaf7b","Type":"ContainerStarted","Data":"037b736e04958f93ebd6345cab2da0d2b65d0a0d939e5c7283a74e18467e7c3a"} Oct 31 21:15:43.337484 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.337354 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" event={"ID":"1b23775a-33a5-47b0-ac74-fb3862912542","Type":"ContainerStarted","Data":"23342e2a936832b22f8ab613a00a5f38fff3fa37c4299db5160ef977d669bb0c"} Oct 31 21:15:43.338908 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.338842 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" event={"ID":"f7d03911-7706-47b8-928e-6ffea21fdb26","Type":"ContainerStarted","Data":"16a63ca010aea713e9ad65816a00f6398257af06dac75fed21210be7279314b5"} Oct 31 21:15:43.342533 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.342499 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w96gb" event={"ID":"9f8d2547-a263-47ff-aaa9-780ea67176a5","Type":"ContainerStarted","Data":"f27aa6f6f5a93e0c95df91f1fb6b7a0ab221c6ede715e47e833d65f0ff601fc8"} Oct 31 21:15:43.342663 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.342551 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w96gb" event={"ID":"9f8d2547-a263-47ff-aaa9-780ea67176a5","Type":"ContainerStarted","Data":"946256b8d18c3f7253e86f5402267c7e80521b1a1326694aab8d6727c85c52fd"} Oct 31 21:15:43.345005 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.344935 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mb4ql" event={"ID":"add13542-315e-4dde-a5fa-728aaf6fdc6c","Type":"ContainerStarted","Data":"15ec11997607027ae5066a249c611525380c2ac036280734b118e4ee59cc7862"} Oct 31 21:15:43.346861 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.346801 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnbr9" event={"ID":"d2a47228-9b40-4833-bb84-970f4782e67b","Type":"ContainerStarted","Data":"fd425a165bedcbef0d9b6e7c11ab02987a2dc413dede5b11e287dd50ce339fff"} Oct 31 21:15:43.351127 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.351099 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7fh65" event={"ID":"4bf7148c-69be-45a9-bd56-7457a3929df8","Type":"ContainerStarted","Data":"2fec51012c5749e9a11fbf588e88cdcd32e103e702c9c66e7ae4f49d4c929b3b"} Oct 31 21:15:43.351247 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.351197 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7fh65" event={"ID":"4bf7148c-69be-45a9-bd56-7457a3929df8","Type":"ContainerStarted","Data":"2dd955e49284c880e75f26845637a762eb0802737a3ea7b18174e9f566869edf"} Oct 31 21:15:43.354206 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.353699 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cps44" event={"ID":"7ed572b5-9b0c-4924-af2f-87703ed1b5e6","Type":"ContainerStarted","Data":"95a325eee05cd52c0520cd8e527a0457cfaa4765c44b0ab71af83bbc161151e7"} Oct 31 21:15:43.354206 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.353741 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cps44" event={"ID":"7ed572b5-9b0c-4924-af2f-87703ed1b5e6","Type":"ContainerStarted","Data":"297bc8e6371cfa9f4f146cee5f2ec511763b3867bdd0ce42e7a9d610c7f5889d"} Oct 31 21:15:43.355812 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.355785 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" event={"ID":"31682289-5cba-480d-88a4-c71e9cd6244b","Type":"ContainerStarted","Data":"5c49701a585b3822378151f7d11449440bd81d2c770c1b0a892bc1c21ec85717"} Oct 31 21:15:43.358402 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.358314 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" event={"ID":"21a28634-04b5-4d1b-bf14-6bdc3b894f2d","Type":"ContainerStarted","Data":"b46c20bb76e6bce8e4e792dc5ccb61793ff1e4d945aeda835f6f484a8123866e"} Oct 31 21:15:43.434777 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.433460 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cps44" podStartSLOduration=325.433441928 podStartE2EDuration="5m25.433441928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:43.432559789 +0000 UTC m=+31.895034605" watchObservedRunningTime="2026-10-31 21:15:43.433441928 +0000 UTC m=+31.895916742" Oct 31 21:15:43.434777 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.434228 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-w96gb" podStartSLOduration=325.434217366 podStartE2EDuration="5m25.434217366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:43.385595932 +0000 UTC m=+31.848070746" watchObservedRunningTime="2026-10-31 21:15:43.434217366 +0000 UTC m=+31.896692192" Oct 31 21:15:43.475697 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.475629 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7fh65" podStartSLOduration=325.475613027 podStartE2EDuration="5m25.475613027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:43.475432994 +0000 UTC m=+31.937907807" watchObservedRunningTime="2026-10-31 21:15:43.475613027 +0000 UTC m=+31.938087844" Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.588484 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.588546 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.588589 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.588629 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.588691 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.588723 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.588770 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.588924 2639 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.588988 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert podName:0bd131db-e6d0-46c6-9a4b-85d3b541a9ec nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.588969549 +0000 UTC m=+34.051444355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sh7vx" (UID: "0bd131db-e6d0-46c6-9a4b-85d3b541a9ec") : secret "networking-console-plugin-cert" not found Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589467 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.589410138 +0000 UTC m=+34.051884943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : configmap references non-existent config key: service-ca.crt Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589549 2639 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589621 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls podName:c01f3957-4a49-41ba-812b-ea2e74d3a7a8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.589577726 +0000 UTC m=+34.052052520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8r9vm" (UID: "c01f3957-4a49-41ba-812b-ea2e74d3a7a8") : secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589683 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589694 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b9c7469-m4qfg: secret "image-registry-tls" not found Oct 31 21:15:43.590019 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589726 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls podName:ce941b3a-da84-4b86-944c-89f9f464a27f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.589714803 +0000 UTC m=+34.052189601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls") pod "image-registry-574b9c7469-m4qfg" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f") : secret "image-registry-tls" not found Oct 31 21:15:43.590618 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589781 2639 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 21:15:43.590618 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589813 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls podName:e49a2d50-dc82-4983-9425-053dfe903ddd nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.589802621 +0000 UTC m=+34.052277426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-k4qxr" (UID: "e49a2d50-dc82-4983-9425-053dfe903ddd") : secret "samples-operator-tls" not found Oct 31 21:15:43.590618 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589861 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:43.590618 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589867 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c58487768-swgvs: secret "image-registry-tls" not found Oct 31 21:15:43.590618 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589885 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls podName:5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.589879787 +0000 UTC m=+34.052354590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls") pod "image-registry-c58487768-swgvs" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b") : secret "image-registry-tls" not found Oct 31 21:15:43.590618 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589922 2639 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 21:15:43.590618 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.589942 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.589933998 +0000 UTC m=+34.052408801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : secret "router-metrics-certs-default" not found Oct 31 21:15:43.690369 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:43.690315 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:43.690564 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.690548 2639 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 21:15:43.690636 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:43.690617 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls podName:b6f98741-1d48-4124-9acb-8c0bf31dbe60 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:45.690599011 +0000 UTC m=+34.153073821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls") pod "dns-default-wj49j" (UID: "b6f98741-1d48-4124-9acb-8c0bf31dbe60") : secret "dns-default-metrics-tls" not found Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.616813 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.616894 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.616933 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.616974 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.617002 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.617044 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.617066 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617212 2639 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617278 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.617257142 +0000 UTC m=+38.079731950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : secret "router-metrics-certs-default" not found Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617456 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617477 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b9c7469-m4qfg: secret "image-registry-tls" not found Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617509 2639 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617521 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls podName:ce941b3a-da84-4b86-944c-89f9f464a27f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.617508333 +0000 UTC m=+38.079983125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls") pod "image-registry-574b9c7469-m4qfg" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f") : secret "image-registry-tls" not found Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617552 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.617534172 +0000 UTC m=+38.080008987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : configmap references non-existent config key: service-ca.crt Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617573 2639 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:45.617645 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617578 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls podName:e49a2d50-dc82-4983-9425-053dfe903ddd nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.617568299 +0000 UTC m=+38.080043091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-k4qxr" (UID: "e49a2d50-dc82-4983-9425-053dfe903ddd") : secret "samples-operator-tls" not found Oct 31 21:15:45.618927 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617585 2639 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 21:15:45.618927 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617598 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls podName:c01f3957-4a49-41ba-812b-ea2e74d3a7a8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.617590988 +0000 UTC m=+38.080065831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8r9vm" (UID: "c01f3957-4a49-41ba-812b-ea2e74d3a7a8") : secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:45.618927 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617613 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert podName:0bd131db-e6d0-46c6-9a4b-85d3b541a9ec nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.617602862 +0000 UTC m=+38.080077653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sh7vx" (UID: "0bd131db-e6d0-46c6-9a4b-85d3b541a9ec") : secret "networking-console-plugin-cert" not found Oct 31 21:15:45.618927 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617642 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:45.618927 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617661 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c58487768-swgvs: secret "image-registry-tls" not found Oct 31 21:15:45.618927 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.617702 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls podName:5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.617690339 +0000 UTC m=+38.080165153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls") pod "image-registry-c58487768-swgvs" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b") : secret "image-registry-tls" not found Oct 31 21:15:45.718636 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.718598 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:45.718817 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.718738 2639 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 21:15:45.718817 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.718817 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls podName:b6f98741-1d48-4124-9acb-8c0bf31dbe60 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.718798539 +0000 UTC m=+38.181273336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls") pod "dns-default-wj49j" (UID: "b6f98741-1d48-4124-9acb-8c0bf31dbe60") : secret "dns-default-metrics-tls" not found Oct 31 21:15:45.819911 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.819840 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:45.819911 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.819956 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:45.820222 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.820106 2639 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 21:15:45.820222 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:45.820154 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:16:17.82014128 +0000 UTC m=+66.282616084 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : secret "metrics-daemon-secret" not found Oct 31 21:15:45.834695 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.834626 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:45.921523 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.920940 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:45.924682 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.924614 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:45.932694 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:45.932442 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:46.053539 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:46.053279 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:46.277927 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:46.277898 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-whzgx"] Oct 31 21:15:46.282265 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:46.282224 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72751047_bc3c_4f2d_9655_e23d5211d346.slice/crio-9a4733ea3a1a5f45ada86ffbae0898784ce87017407cf5e6d1b5cc7aa469b05c WatchSource:0}: Error finding container 9a4733ea3a1a5f45ada86ffbae0898784ce87017407cf5e6d1b5cc7aa469b05c: Status 404 returned error can't find the container with id 9a4733ea3a1a5f45ada86ffbae0898784ce87017407cf5e6d1b5cc7aa469b05c Oct 31 21:15:46.381484 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:46.381443 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-whzgx" event={"ID":"72751047-bc3c-4f2d-9655-e23d5211d346","Type":"ContainerStarted","Data":"9a4733ea3a1a5f45ada86ffbae0898784ce87017407cf5e6d1b5cc7aa469b05c"} Oct 31 21:15:46.383854 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:46.383815 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" event={"ID":"f7d03911-7706-47b8-928e-6ffea21fdb26","Type":"ContainerStarted","Data":"05a7ccb20b715279b09970382b28f498eb1e2bb3854001a15e6ed2597ecce410"} Oct 31 21:15:46.431564 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:46.427800 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" podStartSLOduration=300.775754747 podStartE2EDuration="5m3.427777048s" totalImagesPullingTime="2.652022301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:46.426537577 +0000 UTC m=+34.889012391" watchObservedRunningTime="2026-10-31 21:15:46.427777048 +0000 UTC m=+34.890251863" Oct 31 21:15:46.431564 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:46.430538 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tpl7t"] Oct 31 21:15:46.438243 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:46.438204 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc929e38e_147c_41f9_ad34_b0af48044e09.slice/crio-078a12eed37966aef9fb378b99a4a30c60da7a6072fd39c38fab49c2d3824d04 WatchSource:0}: Error finding container 078a12eed37966aef9fb378b99a4a30c60da7a6072fd39c38fab49c2d3824d04: Status 404 returned error can't find the container with id 078a12eed37966aef9fb378b99a4a30c60da7a6072fd39c38fab49c2d3824d04 Oct 31 21:15:47.392468 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:47.392438 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tpl7t" event={"ID":"c929e38e-147c-41f9-ad34-b0af48044e09","Type":"ContainerStarted","Data":"078a12eed37966aef9fb378b99a4a30c60da7a6072fd39c38fab49c2d3824d04"} Oct 31 21:15:47.395190 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:47.395165 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnbr9" event={"ID":"d2a47228-9b40-4833-bb84-970f4782e67b","Type":"ContainerStarted","Data":"43fc9c1ba32e4402ba3e187e507f3f10a97462eefa6747b2a0ac6c7d0fd4254e"} Oct 31 21:15:48.173042 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.172956 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-bnbr9" podStartSLOduration=300.843811227 podStartE2EDuration="5m5.172936036s" totalImagesPullingTime="4.329124809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:47.440876829 +0000 UTC m=+35.903351642" watchObservedRunningTime="2026-10-31 21:15:48.172936036 +0000 UTC m=+36.635410850" Oct 31 21:15:48.174558 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.174530 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dhxqp"] Oct 31 21:15:48.194316 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.194270 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.200409 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.199819 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dhxqp"] Oct 31 21:15:48.210562 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.210517 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 21:15:48.210929 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.210904 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 21:15:48.211051 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.211037 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-866d2\"" Oct 31 21:15:48.261263 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.261222 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-vk6tn-jprcv"] Oct 31 21:15:48.275260 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.275224 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.298592 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.296701 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vk6tn-jprcv"] Oct 31 21:15:48.347851 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.347805 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.348034 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.347961 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.348034 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.347992 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whfjx\" (UniqueName: \"kubernetes.io/projected/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-api-access-whfjx\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.348034 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.348025 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7245fb80-34d5-471b-984e-7fbfe2b1113c-crio-socket\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.348205 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.348093 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7245fb80-34d5-471b-984e-7fbfe2b1113c-data-volume\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.411064 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.411021 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mb4ql" event={"ID":"add13542-315e-4dde-a5fa-728aaf6fdc6c","Type":"ContainerStarted","Data":"3005c1d72a8c95a9049db52b30b587354469952d44b6f07eb0f811c89f1ae701"} Oct 31 21:15:48.414093 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.414056 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" event={"ID":"31682289-5cba-480d-88a4-c71e9cd6244b","Type":"ContainerStarted","Data":"e285ee5c03acfccd850fce84974166a73e70c55a8c44d1fb22ad2861554f963c"} Oct 31 21:15:48.417445 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.416823 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" event={"ID":"21a28634-04b5-4d1b-bf14-6bdc3b894f2d","Type":"ContainerStarted","Data":"e135dc16a77e1e00689eff2607c2eb2820145495534eba5724c5b72afe8815d6"} Oct 31 21:15:48.421475 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.420088 2639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/0.log" Oct 31 21:15:48.421475 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.420133 2639 generic.go:356] "Generic (PLEG): container finished" podID="1b23775a-33a5-47b0-ac74-fb3862912542" containerID="bc6fbb160dfb367724db515a4616e4ae1c1395b6ab0468f5bb87ec408f23e1b4" exitCode=255 Oct 31 21:15:48.421475 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.420988 2639 scope.go:122] "RemoveContainer" containerID="bc6fbb160dfb367724db515a4616e4ae1c1395b6ab0468f5bb87ec408f23e1b4" Oct 31 21:15:48.421475 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.421403 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" event={"ID":"1b23775a-33a5-47b0-ac74-fb3862912542","Type":"ContainerDied","Data":"bc6fbb160dfb367724db515a4616e4ae1c1395b6ab0468f5bb87ec408f23e1b4"} Oct 31 21:15:48.448761 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.448711 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7245fb80-34d5-471b-984e-7fbfe2b1113c-data-volume\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.448950 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.448798 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.448950 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.448852 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.449051 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.448963 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.449051 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.448996 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnm6\" (UniqueName: \"kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.449051 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.449028 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whfjx\" (UniqueName: \"kubernetes.io/projected/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-api-access-whfjx\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.449191 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.449064 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7245fb80-34d5-471b-984e-7fbfe2b1113c-crio-socket\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.449191 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.449126 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.449191 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.449159 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.449519 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:48.449497 2639 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 21:15:48.449617 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:48.449572 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:48.949550431 +0000 UTC m=+37.412025223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : secret "insights-runtime-extractor-tls" not found Oct 31 21:15:48.449921 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:48.449880 2639 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 21:15:48.450001 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:48.449931 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:48.949915603 +0000 UTC m=+37.412390397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : configmap "kube-rbac-proxy" not found Oct 31 21:15:48.450060 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.450001 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7245fb80-34d5-471b-984e-7fbfe2b1113c-crio-socket\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.450693 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.450672 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7245fb80-34d5-471b-984e-7fbfe2b1113c-data-volume\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.483011 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.482975 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whfjx\" (UniqueName: \"kubernetes.io/projected/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-api-access-whfjx\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.497922 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.497262 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mb4ql" podStartSLOduration=2.125326909 podStartE2EDuration="7.497237205s" totalImagesPullingTime="5.371910296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:48.455084545 +0000 UTC m=+36.917559358" watchObservedRunningTime="2026-10-31 21:15:48.497237205 +0000 UTC m=+36.959712018" Oct 31 21:15:48.542438 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.541480 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" podStartSLOduration=295.584359264 podStartE2EDuration="5m0.541463063s" totalImagesPullingTime="4.957103799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:48.498493458 +0000 UTC m=+36.960968272" watchObservedRunningTime="2026-10-31 21:15:48.541463063 +0000 UTC m=+37.003937876" Oct 31 21:15:48.550801 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.550711 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnm6\" (UniqueName: \"kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.550945 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.550842 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.550945 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.550910 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.551083 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.551067 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.552398 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.552368 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.553796 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.553763 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.554669 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.554643 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.579531 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.579443 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnm6\" (UniqueName: \"kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.583875 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.581778 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" podStartSLOduration=300.60096599 podStartE2EDuration="5m5.581759784s" totalImagesPullingTime="4.980793794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:48.581458358 +0000 UTC m=+37.043933171" watchObservedRunningTime="2026-10-31 21:15:48.581759784 +0000 UTC m=+37.044234598" Oct 31 21:15:48.623509 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.623466 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:48.838545 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.838454 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd"] Oct 31 21:15:48.856723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.856686 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x"] Oct 31 21:15:48.856905 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.856832 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" Oct 31 21:15:48.873157 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.873131 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:48.873283 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.873264 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 21:15:48.873385 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.873293 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ngrs9\"" Oct 31 21:15:48.882827 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.882798 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd"] Oct 31 21:15:48.882991 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.882845 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x"] Oct 31 21:15:48.882991 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.882959 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" Oct 31 21:15:48.956559 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.954641 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.956559 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.954738 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rd6p\" (UniqueName: \"kubernetes.io/projected/4873e537-31a8-4e6d-86bd-c54024d94c39-kube-api-access-4rd6p\") pod \"migrator-66df5d496b-g7thd\" (UID: \"4873e537-31a8-4e6d-86bd-c54024d94c39\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" Oct 31 21:15:48.956559 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.954853 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:48.956559 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:48.954959 2639 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 21:15:48.956559 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:48.955022 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.955004223 +0000 UTC m=+38.417479020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : configmap "kube-rbac-proxy" not found Oct 31 21:15:48.956559 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:48.955105 2639 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 21:15:48.956559 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:48.955169 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:49.955150029 +0000 UTC m=+38.417624843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : secret "insights-runtime-extractor-tls" not found Oct 31 21:15:48.956559 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:48.956485 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-vk6tn-jprcv"] Oct 31 21:15:49.056206 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.056155 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rd6p\" (UniqueName: \"kubernetes.io/projected/4873e537-31a8-4e6d-86bd-c54024d94c39-kube-api-access-4rd6p\") pod \"migrator-66df5d496b-g7thd\" (UID: \"4873e537-31a8-4e6d-86bd-c54024d94c39\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" Oct 31 21:15:49.057141 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.057095 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5f5z\" (UniqueName: \"kubernetes.io/projected/25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5-kube-api-access-p5f5z\") pod \"migrator-66df5d496b-qrc2x\" (UID: \"25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" Oct 31 21:15:49.080813 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.080773 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rd6p\" (UniqueName: \"kubernetes.io/projected/4873e537-31a8-4e6d-86bd-c54024d94c39-kube-api-access-4rd6p\") pod \"migrator-66df5d496b-g7thd\" (UID: \"4873e537-31a8-4e6d-86bd-c54024d94c39\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" Oct 31 21:15:49.114233 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:49.114193 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29050b65_73ea_446c_98b6_76ce5438f0c9.slice/crio-7c0d5c17583db4e70c67c6119379ee93e013f8fc26f49b2ec07eb16f785787f0 WatchSource:0}: Error finding container 7c0d5c17583db4e70c67c6119379ee93e013f8fc26f49b2ec07eb16f785787f0: Status 404 returned error can't find the container with id 7c0d5c17583db4e70c67c6119379ee93e013f8fc26f49b2ec07eb16f785787f0 Oct 31 21:15:49.117297 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.117274 2639 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8qdzg"] Oct 31 21:15:49.140386 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.140340 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8qdzg"] Oct 31 21:15:49.140565 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.140497 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.155723 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.155691 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 21:15:49.156091 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.156071 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 21:15:49.156267 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.156089 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 21:15:49.156352 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.156123 2639 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 21:15:49.156352 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.156169 2639 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qtvpr\"" Oct 31 21:15:49.158304 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.158274 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5f5z\" (UniqueName: \"kubernetes.io/projected/25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5-kube-api-access-p5f5z\") pod \"migrator-66df5d496b-qrc2x\" (UID: \"25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" Oct 31 21:15:49.184455 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.184423 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5f5z\" (UniqueName: \"kubernetes.io/projected/25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5-kube-api-access-p5f5z\") pod \"migrator-66df5d496b-qrc2x\" (UID: \"25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" Oct 31 21:15:49.189672 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.189472 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" Oct 31 21:15:49.212224 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.212155 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" Oct 31 21:15:49.260248 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.259872 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/595edb9d-3370-41bd-82c3-af8df664ad90-signing-cabundle\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.260248 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.260107 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j274g\" (UniqueName: \"kubernetes.io/projected/595edb9d-3370-41bd-82c3-af8df664ad90-kube-api-access-j274g\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.260248 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.260167 2639 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/595edb9d-3370-41bd-82c3-af8df664ad90-signing-key\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.361868 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.361832 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j274g\" (UniqueName: \"kubernetes.io/projected/595edb9d-3370-41bd-82c3-af8df664ad90-kube-api-access-j274g\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.362083 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.361916 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/595edb9d-3370-41bd-82c3-af8df664ad90-signing-key\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.362083 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.361962 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/595edb9d-3370-41bd-82c3-af8df664ad90-signing-cabundle\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.363105 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.362936 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/595edb9d-3370-41bd-82c3-af8df664ad90-signing-cabundle\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.365286 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.365216 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/595edb9d-3370-41bd-82c3-af8df664ad90-signing-key\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.387009 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.386973 2639 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j274g\" (UniqueName: \"kubernetes.io/projected/595edb9d-3370-41bd-82c3-af8df664ad90-kube-api-access-j274g\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.429359 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.428894 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" event={"ID":"e6924bea-83c3-443f-a95d-581edf9aaf7b","Type":"ContainerStarted","Data":"e9edb791cf4a7e520262afac148166c2d58239067a36f4e141ee3c816cf9ccaa"} Oct 31 21:15:49.431523 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.431477 2639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/1.log" Oct 31 21:15:49.432597 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.432571 2639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/0.log" Oct 31 21:15:49.432717 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.432613 2639 generic.go:356] "Generic (PLEG): container finished" podID="1b23775a-33a5-47b0-ac74-fb3862912542" containerID="0728d08f5fd76e870a74999860ecc1ce866ab9547dabb2fedb55a89978d2fab1" exitCode=255 Oct 31 21:15:49.432717 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.432706 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" event={"ID":"1b23775a-33a5-47b0-ac74-fb3862912542","Type":"ContainerDied","Data":"0728d08f5fd76e870a74999860ecc1ce866ab9547dabb2fedb55a89978d2fab1"} Oct 31 21:15:49.432813 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.432763 2639 scope.go:122] "RemoveContainer" containerID="bc6fbb160dfb367724db515a4616e4ae1c1395b6ab0468f5bb87ec408f23e1b4" Oct 31 21:15:49.433239 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.433217 2639 scope.go:122] "RemoveContainer" containerID="0728d08f5fd76e870a74999860ecc1ce866ab9547dabb2fedb55a89978d2fab1" Oct 31 21:15:49.433523 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.433493 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cg8tz_openshift-console-operator(1b23775a-33a5-47b0-ac74-fb3862912542)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" podUID="1b23775a-33a5-47b0-ac74-fb3862912542" Oct 31 21:15:49.436194 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.436161 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" event={"ID":"29050b65-73ea-446c-98b6-76ce5438f0c9","Type":"ContainerStarted","Data":"6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51"} Oct 31 21:15:49.436294 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.436216 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" event={"ID":"29050b65-73ea-446c-98b6-76ce5438f0c9","Type":"ContainerStarted","Data":"7c0d5c17583db4e70c67c6119379ee93e013f8fc26f49b2ec07eb16f785787f0"} Oct 31 21:15:49.470893 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.470839 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" podStartSLOduration=293.009207251 podStartE2EDuration="4m59.470822721s" totalImagesPullingTime="6.46161547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:49.469578657 +0000 UTC m=+37.932053471" watchObservedRunningTime="2026-10-31 21:15:49.470822721 +0000 UTC m=+37.933297534" Oct 31 21:15:49.477660 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.477604 2639 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:49.538528 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.538491 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd"] Oct 31 21:15:49.542687 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:49.542645 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4873e537_31a8_4e6d_86bd_c54024d94c39.slice/crio-6cc24fa0fe204e03da7301b3ec40e6f21239820cd57a5cbc31c8ec56643306ef WatchSource:0}: Error finding container 6cc24fa0fe204e03da7301b3ec40e6f21239820cd57a5cbc31c8ec56643306ef: Status 404 returned error can't find the container with id 6cc24fa0fe204e03da7301b3ec40e6f21239820cd57a5cbc31c8ec56643306ef Oct 31 21:15:49.569720 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.567855 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" podStartSLOduration=1.567835726 podStartE2EDuration="1.567835726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:49.566665684 +0000 UTC m=+38.029140501" watchObservedRunningTime="2026-10-31 21:15:49.567835726 +0000 UTC m=+38.030310540" Oct 31 21:15:49.582455 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.582312 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x"] Oct 31 21:15:49.585186 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:49.585147 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25bcf1fd_b559_4c7c_b6af_22ff9d86e6a5.slice/crio-66d4bcc4c3829affdf35b569dba49e789ff46376f4feaf1461e095084f0a3d9b WatchSource:0}: Error finding container 66d4bcc4c3829affdf35b569dba49e789ff46376f4feaf1461e095084f0a3d9b: Status 404 returned error can't find the container with id 66d4bcc4c3829affdf35b569dba49e789ff46376f4feaf1461e095084f0a3d9b Oct 31 21:15:49.665988 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.665906 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:49.665988 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.665969 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:49.666164 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.666018 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:49.666164 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.666089 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:49.666164 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.666159 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:49.666283 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.666182 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:49.666283 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.666231 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:49.666478 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666457 2639 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 21:15:49.666550 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666519 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert podName:0bd131db-e6d0-46c6-9a4b-85d3b541a9ec nodeName:}" failed. No retries permitted until 2026-10-31 21:15:57.666505259 +0000 UTC m=+46.128980050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sh7vx" (UID: "0bd131db-e6d0-46c6-9a4b-85d3b541a9ec") : secret "networking-console-plugin-cert" not found Oct 31 21:15:49.666614 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666582 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:57.666575046 +0000 UTC m=+46.129049842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : configmap references non-existent config key: service-ca.crt Oct 31 21:15:49.666666 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666627 2639 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:49.666666 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666649 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls podName:c01f3957-4a49-41ba-812b-ea2e74d3a7a8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:57.666643276 +0000 UTC m=+46.129118067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8r9vm" (UID: "c01f3957-4a49-41ba-812b-ea2e74d3a7a8") : secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:49.666749 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666687 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:49.666749 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666695 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-574b9c7469-m4qfg: secret "image-registry-tls" not found Oct 31 21:15:49.666749 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666719 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls podName:ce941b3a-da84-4b86-944c-89f9f464a27f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:57.666712367 +0000 UTC m=+46.129187173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls") pod "image-registry-574b9c7469-m4qfg" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f") : secret "image-registry-tls" not found Oct 31 21:15:49.666879 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666757 2639 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 21:15:49.666879 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666775 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls podName:e49a2d50-dc82-4983-9425-053dfe903ddd nodeName:}" failed. No retries permitted until 2026-10-31 21:15:57.666769791 +0000 UTC m=+46.129244581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-k4qxr" (UID: "e49a2d50-dc82-4983-9425-053dfe903ddd") : secret "samples-operator-tls" not found Oct 31 21:15:49.666879 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666815 2639 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 21:15:49.666879 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666822 2639 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c58487768-swgvs: secret "image-registry-tls" not found Oct 31 21:15:49.666879 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666858 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls podName:5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b nodeName:}" failed. No retries permitted until 2026-10-31 21:15:57.666852568 +0000 UTC m=+46.129327359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls") pod "image-registry-c58487768-swgvs" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b") : secret "image-registry-tls" not found Oct 31 21:15:49.667044 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666901 2639 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 21:15:49.667044 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.666919 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs podName:934a6c09-d6c4-4ae2-afbf-3bc19cc948c8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:57.666914556 +0000 UTC m=+46.129389347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs") pod "router-default-68d4c65fcc-47wdn" (UID: "934a6c09-d6c4-4ae2-afbf-3bc19cc948c8") : secret "router-metrics-certs-default" not found Oct 31 21:15:49.767308 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.767259 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:49.767557 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.767525 2639 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 21:15:49.767698 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.767598 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls podName:b6f98741-1d48-4124-9acb-8c0bf31dbe60 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:57.76758285 +0000 UTC m=+46.230057679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls") pod "dns-default-wj49j" (UID: "b6f98741-1d48-4124-9acb-8c0bf31dbe60") : secret "dns-default-metrics-tls" not found Oct 31 21:15:49.815269 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.815225 2639 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8qdzg"] Oct 31 21:15:49.817771 ip-10-0-131-21 kubenswrapper[2639]: W1031 21:15:49.817738 2639 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595edb9d_3370_41bd_82c3_af8df664ad90.slice/crio-a5b9999dedd757a42d09d4f770e53a67a8585c6399d7c6d68486664b6ddcfd9a WatchSource:0}: Error finding container a5b9999dedd757a42d09d4f770e53a67a8585c6399d7c6d68486664b6ddcfd9a: Status 404 returned error can't find the container with id a5b9999dedd757a42d09d4f770e53a67a8585c6399d7c6d68486664b6ddcfd9a Oct 31 21:15:49.969727 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.969529 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:49.969727 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:49.969633 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:49.969727 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.969702 2639 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 21:15:49.969985 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.969782 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:51.969765813 +0000 UTC m=+40.432240605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : secret "insights-runtime-extractor-tls" not found Oct 31 21:15:49.970065 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.969707 2639 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 21:15:49.970192 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:49.970108 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:51.970093178 +0000 UTC m=+40.432567982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : configmap "kube-rbac-proxy" not found Oct 31 21:15:50.447677 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:50.447638 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" event={"ID":"595edb9d-3370-41bd-82c3-af8df664ad90","Type":"ContainerStarted","Data":"8072b9d9acb0b6b155271ca0dd93984ba4483ff57fa7e2fa6324d620ebdf5283"} Oct 31 21:15:50.448145 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:50.447703 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" event={"ID":"595edb9d-3370-41bd-82c3-af8df664ad90","Type":"ContainerStarted","Data":"a5b9999dedd757a42d09d4f770e53a67a8585c6399d7c6d68486664b6ddcfd9a"} Oct 31 21:15:50.451589 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:50.451506 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" event={"ID":"25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5","Type":"ContainerStarted","Data":"66d4bcc4c3829affdf35b569dba49e789ff46376f4feaf1461e095084f0a3d9b"} Oct 31 21:15:50.453895 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:50.453827 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" event={"ID":"4873e537-31a8-4e6d-86bd-c54024d94c39","Type":"ContainerStarted","Data":"6cc24fa0fe204e03da7301b3ec40e6f21239820cd57a5cbc31c8ec56643306ef"} Oct 31 21:15:50.459384 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:50.458983 2639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/1.log" Oct 31 21:15:50.460438 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:50.459902 2639 scope.go:122] "RemoveContainer" containerID="0728d08f5fd76e870a74999860ecc1ce866ab9547dabb2fedb55a89978d2fab1" Oct 31 21:15:50.460438 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:50.460097 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cg8tz_openshift-console-operator(1b23775a-33a5-47b0-ac74-fb3862912542)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" podUID="1b23775a-33a5-47b0-ac74-fb3862912542" Oct 31 21:15:50.492797 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:50.492731 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" podStartSLOduration=1.4927109619999999 podStartE2EDuration="1.492710962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:50.491423282 +0000 UTC m=+38.953898101" watchObservedRunningTime="2026-10-31 21:15:50.492710962 +0000 UTC m=+38.955185775" Oct 31 21:15:51.468751 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:51.468709 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" event={"ID":"4873e537-31a8-4e6d-86bd-c54024d94c39","Type":"ContainerStarted","Data":"3ce66e9177c85ce2ec4f425420e45a8a5b1eab48b4f7688d069803208c467193"} Oct 31 21:15:51.472697 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:51.472668 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tpl7t" event={"ID":"c929e38e-147c-41f9-ad34-b0af48044e09","Type":"ContainerStarted","Data":"cf6c0d3f7892b38e6bc78c71e8ffc6636217a06bcdec6b165424b8f2c6572200"} Oct 31 21:15:51.472842 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:51.472735 2639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:51.531350 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:51.531283 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tpl7t" podStartSLOduration=35.578658157 podStartE2EDuration="39.531266014s" totalImagesPullingTime="3.952607857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:51.530447939 +0000 UTC m=+39.992922753" watchObservedRunningTime="2026-10-31 21:15:51.531266014 +0000 UTC m=+39.993740828" Oct 31 21:15:51.994881 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:51.994837 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:51.995079 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:51.994958 2639 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 21:15:51.995079 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:51.994982 2639 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:51.995079 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:51.995034 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:55.995018153 +0000 UTC m=+44.457492966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : configmap "kube-rbac-proxy" not found Oct 31 21:15:51.995214 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:51.995109 2639 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 21:15:51.995214 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:51.995164 2639 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:55.99514926 +0000 UTC m=+44.457624064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : secret "insights-runtime-extractor-tls" not found Oct 31 21:15:52.048088 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:52.048047 2639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vx79w_6805be97-0a2e-490c-a50c-6700f15c4abc/dns-node-resolver/0.log" Oct 31 21:15:52.291589 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:52.291500 2639 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:52.291589 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:52.291574 2639 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:52.292040 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:52.292022 2639 scope.go:122] "RemoveContainer" containerID="0728d08f5fd76e870a74999860ecc1ce866ab9547dabb2fedb55a89978d2fab1" Oct 31 21:15:52.292282 ip-10-0-131-21 kubenswrapper[2639]: E1031 21:15:52.292255 2639 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cg8tz_openshift-console-operator(1b23775a-33a5-47b0-ac74-fb3862912542)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" podUID="1b23775a-33a5-47b0-ac74-fb3862912542" Oct 31 21:15:52.478788 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:52.478755 2639 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" event={"ID":"25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5","Type":"ContainerStarted","Data":"cac4134f467f0c39b0278fe61fc3c93e6afbc6f7cf3848fc4d1053020b3697e2"} Oct 31 21:15:52.521682 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:52.521627 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" podStartSLOduration=2.715907203 podStartE2EDuration="4.521610062s" totalImagesPullingTime="1.805702859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:52.520173636 +0000 UTC m=+40.982648446" watchObservedRunningTime="2026-10-31 21:15:52.521610062 +0000 UTC m=+40.984084933" Oct 31 21:15:52.563401 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:52.563271 2639 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" podStartSLOduration=2.564492671 podStartE2EDuration="4.56325082s" totalImagesPullingTime="1.998758149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:52.561823517 +0000 UTC m=+41.024298333" watchObservedRunningTime="2026-10-31 21:15:52.56325082 +0000 UTC m=+41.025725636" Oct 31 21:15:53.048010 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:53.047979 2639 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-42vxn_672a2ce2-d45d-4f0c-a75a-b018105640aa/node-ca/0.log" Oct 31 21:15:53.253625 ip-10-0-131-21 kubenswrapper[2639]: I1031 21:15:53.253456 2639 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:53.253631 ip-10-0-131-21 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 21:15:53.263780 ip-10-0-131-21 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 21:15:53.263961 ip-10-0-131-21 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 21:15:53.264013 ip-10-0-131-21 systemd[1]: kubelet.service: Consumed 3.470s CPU time. Oct 31 21:15:53.275111 ip-10-0-131-21 systemd[1]: Starting Kubernetes Kubelet... Oct 31 21:15:53.372296 ip-10-0-131-21 kubenswrapper[5188]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:53.372296 ip-10-0-131-21 kubenswrapper[5188]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 21:15:53.372296 ip-10-0-131-21 kubenswrapper[5188]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:53.372296 ip-10-0-131-21 kubenswrapper[5188]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:53.374515 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374401 5188 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:53.374515 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374514 5188 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:53.374515 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374519 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374522 5188 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374525 5188 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374528 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374533 5188 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374537 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374540 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374544 5188 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374547 5188 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374550 5188 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374553 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374555 5188 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374558 5188 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374561 5188 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374564 5188 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374568 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374571 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374574 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374576 5188 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:53.374627 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374579 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374587 5188 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374591 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374594 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374596 5188 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374599 5188 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374602 5188 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374605 5188 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374608 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374610 5188 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374613 5188 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374616 5188 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374618 5188 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374621 5188 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374623 5188 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374626 5188 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374629 5188 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374632 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374635 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374637 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:53.375111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374640 5188 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374643 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374647 5188 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374649 5188 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374652 5188 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374654 5188 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374657 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374660 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374662 5188 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374665 5188 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374668 5188 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374671 5188 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374673 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374676 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374679 5188 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374681 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374684 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374687 5188 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374690 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374693 5188 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:53.375668 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374696 5188 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374700 5188 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374704 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374707 5188 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374710 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374713 5188 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374715 5188 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374718 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374721 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374723 5188 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374726 5188 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374728 5188 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374731 5188 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374734 5188 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374737 5188 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374746 5188 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374750 5188 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374753 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374756 5188 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:53.376171 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374758 5188 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374761 5188 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374764 5188 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374767 5188 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374769 5188 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374772 5188 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374775 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374778 5188 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374780 5188 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374783 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374786 5188 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374789 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374792 5188 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374794 5188 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374797 5188 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374799 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374802 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374805 5188 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374807 5188 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374810 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:53.376659 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374812 5188 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374815 5188 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374818 5188 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374820 5188 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374823 5188 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.374826 5188 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375662 5188 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375670 5188 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375673 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375677 5188 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375681 5188 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375686 5188 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375689 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375691 5188 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375694 5188 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375697 5188 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375700 5188 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375703 5188 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375706 5188 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375709 5188 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:53.377176 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375711 5188 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375714 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375717 5188 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375719 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375722 5188 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375725 5188 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375728 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375731 5188 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375734 5188 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375736 5188 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375739 5188 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375741 5188 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375744 5188 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375746 5188 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375749 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375751 5188 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375754 5188 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375756 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375759 5188 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375762 5188 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:53.377709 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375765 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375768 5188 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375770 5188 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375773 5188 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375776 5188 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375778 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375781 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375785 5188 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375787 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375790 5188 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375792 5188 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375795 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375797 5188 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375800 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375803 5188 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375806 5188 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375809 5188 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375811 5188 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375814 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:53.378275 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375817 5188 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375819 5188 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375822 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375825 5188 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375827 5188 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375830 5188 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375832 5188 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375835 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375837 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375840 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375843 5188 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375846 5188 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375849 5188 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375851 5188 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375854 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375856 5188 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375859 5188 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375862 5188 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375864 5188 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375867 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:53.378781 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375869 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375872 5188 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375875 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375877 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375880 5188 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375882 5188 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375885 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375891 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375893 5188 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375896 5188 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375899 5188 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375901 5188 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375904 5188 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375906 5188 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375909 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375911 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375914 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375916 5188 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375921 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375923 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:53.379284 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375926 5188 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375929 5188 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375932 5188 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375934 5188 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375937 5188 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375939 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375942 5188 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375944 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375947 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375950 5188 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375953 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375957 5188 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.375961 5188 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376474 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376480 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376482 5188 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376486 5188 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376489 5188 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376493 5188 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:53.379837 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376499 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376502 5188 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376505 5188 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376508 5188 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376511 5188 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376514 5188 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376517 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376519 5188 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376522 5188 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376525 5188 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376527 5188 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376531 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376534 5188 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376536 5188 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376539 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376542 5188 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376544 5188 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376547 5188 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376550 5188 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376552 5188 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:53.380312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376555 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376558 5188 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376561 5188 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376564 5188 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376566 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376569 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376572 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376574 5188 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376577 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376580 5188 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376583 5188 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376585 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376589 5188 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376592 5188 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376594 5188 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376597 5188 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376599 5188 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376602 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376604 5188 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:53.381112 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376607 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376609 5188 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376612 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376615 5188 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376618 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376621 5188 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376623 5188 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376626 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376628 5188 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376631 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376634 5188 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376636 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376639 5188 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376641 5188 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376644 5188 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376646 5188 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376649 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376652 5188 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376655 5188 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376658 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:53.381968 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376661 5188 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376664 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376666 5188 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376671 5188 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376674 5188 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376679 5188 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376682 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376685 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376687 5188 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376690 5188 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376692 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376695 5188 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376697 5188 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376700 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376702 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376705 5188 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376708 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376711 5188 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376714 5188 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376716 5188 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:53.382877 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376719 5188 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376721 5188 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376724 5188 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376727 5188 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376730 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376733 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376735 5188 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376738 5188 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376741 5188 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376744 5188 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376746 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376749 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376751 5188 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376754 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376756 5188 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376759 5188 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376761 5188 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376765 5188 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376767 5188 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:53.383796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376770 5188 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.376772 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376875 5188 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376883 5188 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376890 5188 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376895 5188 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376901 5188 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376904 5188 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376909 5188 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376914 5188 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376918 5188 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376921 5188 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376925 5188 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376928 5188 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376931 5188 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376935 5188 flags.go:64] FLAG: --cgroup-root="" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376938 5188 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376941 5188 flags.go:64] FLAG: --client-ca-file="" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376944 5188 flags.go:64] FLAG: --cloud-provider="external" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376947 5188 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376951 5188 flags.go:64] FLAG: --cluster-domain="" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376954 5188 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376958 5188 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376962 5188 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 21:15:53.384660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376966 5188 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376970 5188 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376973 5188 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376976 5188 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376980 5188 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376983 5188 flags.go:64] FLAG: --contention-profiling="false" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376986 5188 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376989 5188 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376993 5188 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.376996 5188 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377000 5188 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377003 5188 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377006 5188 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377009 5188 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377012 5188 flags.go:64] FLAG: --enable-server="true" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377016 5188 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377020 5188 flags.go:64] FLAG: --event-burst="100" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377026 5188 flags.go:64] FLAG: --event-qps="50" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377029 5188 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377032 5188 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377036 5188 flags.go:64] FLAG: --eviction-hard="" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377040 5188 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377043 5188 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377046 5188 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377049 5188 flags.go:64] FLAG: --eviction-soft="" Oct 31 21:15:53.385730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377053 5188 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377056 5188 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377059 5188 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377062 5188 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377065 5188 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377069 5188 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377072 5188 flags.go:64] FLAG: --feature-gates="" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377077 5188 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377080 5188 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377084 5188 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377087 5188 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377090 5188 flags.go:64] FLAG: --healthz-port="10248" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377093 5188 flags.go:64] FLAG: --help="false" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377096 5188 flags.go:64] FLAG: --hostname-override="ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377100 5188 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377103 5188 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377106 5188 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377109 5188 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377112 5188 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377115 5188 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377118 5188 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377121 5188 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377124 5188 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 21:15:53.386845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377127 5188 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377131 5188 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377134 5188 flags.go:64] FLAG: --kube-reserved="" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377137 5188 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377142 5188 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377145 5188 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377148 5188 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377151 5188 flags.go:64] FLAG: --lock-file="" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377154 5188 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377157 5188 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377160 5188 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377166 5188 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377170 5188 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377173 5188 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377176 5188 flags.go:64] FLAG: --logging-format="text" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377179 5188 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377182 5188 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377187 5188 flags.go:64] FLAG: --manifest-url="" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377190 5188 flags.go:64] FLAG: --manifest-url-header="" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377195 5188 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377198 5188 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377201 5188 flags.go:64] FLAG: --max-pods="110" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377204 5188 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377207 5188 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377210 5188 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 21:15:53.387946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377213 5188 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377216 5188 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377220 5188 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377223 5188 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377229 5188 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377232 5188 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377236 5188 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377239 5188 flags.go:64] FLAG: --pod-cidr="" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377243 5188 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377245 5188 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377249 5188 flags.go:64] FLAG: --pods-per-core="0" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377253 5188 flags.go:64] FLAG: --port="10250" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377256 5188 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377259 5188 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02e5be499730184ae" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377262 5188 flags.go:64] FLAG: --qos-reserved="" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377265 5188 flags.go:64] FLAG: --read-only-port="10255" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377268 5188 flags.go:64] FLAG: --register-node="true" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377271 5188 flags.go:64] FLAG: --register-with-taints="" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377275 5188 flags.go:64] FLAG: --registry-burst="10" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377278 5188 flags.go:64] FLAG: --registry-qps="5" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377281 5188 flags.go:64] FLAG: --reserved-cpus="" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377284 5188 flags.go:64] FLAG: --reserved-memory="" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377288 5188 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377292 5188 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377295 5188 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377299 5188 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 21:15:53.388996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377301 5188 flags.go:64] FLAG: --runonce="false" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377304 5188 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377308 5188 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377311 5188 flags.go:64] FLAG: --seccomp-default="false" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377314 5188 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377316 5188 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377334 5188 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377338 5188 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377341 5188 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377344 5188 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377347 5188 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377350 5188 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377353 5188 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377357 5188 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377360 5188 flags.go:64] FLAG: --system-cgroups="" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377363 5188 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377369 5188 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377373 5188 flags.go:64] FLAG: --tls-cert-file="" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377376 5188 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377380 5188 flags.go:64] FLAG: --tls-min-version="" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377383 5188 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377386 5188 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377389 5188 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377392 5188 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377395 5188 flags.go:64] FLAG: --v="2" Oct 31 21:15:53.390049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377400 5188 flags.go:64] FLAG: --version="false" Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377404 5188 flags.go:64] FLAG: --vmodule="" Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377408 5188 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377412 5188 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377563 5188 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377568 5188 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377571 5188 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377574 5188 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377577 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377580 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377582 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377585 5188 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377587 5188 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377590 5188 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377593 5188 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377596 5188 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377598 5188 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377601 5188 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377603 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:53.390891 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377606 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377609 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377611 5188 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377614 5188 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377616 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377621 5188 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377623 5188 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377626 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377628 5188 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377631 5188 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377634 5188 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377636 5188 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377639 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377641 5188 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377644 5188 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377646 5188 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377649 5188 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377651 5188 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377654 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377656 5188 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:53.391519 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377659 5188 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377662 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377665 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377668 5188 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377670 5188 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377672 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377675 5188 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377678 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377680 5188 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377683 5188 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377685 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377688 5188 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377690 5188 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377694 5188 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377697 5188 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377700 5188 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377704 5188 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377709 5188 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377712 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:53.392142 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377714 5188 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377718 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377720 5188 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377724 5188 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377726 5188 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377729 5188 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377731 5188 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377734 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377737 5188 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377739 5188 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377742 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377744 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377747 5188 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377750 5188 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377752 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377755 5188 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377759 5188 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377764 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377767 5188 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377771 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:53.392995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377773 5188 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377776 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377779 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377782 5188 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377784 5188 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377787 5188 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377791 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377794 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377796 5188 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377799 5188 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377802 5188 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377805 5188 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377808 5188 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377810 5188 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377813 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377816 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377818 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377821 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377823 5188 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:53.393526 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377826 5188 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377828 5188 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377831 5188 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377834 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377837 5188 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377839 5188 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377842 5188 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377844 5188 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377847 5188 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377850 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377853 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377855 5188 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377858 5188 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.377882 5188 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377918 5188 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377923 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:53.394095 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377926 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377928 5188 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377931 5188 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377934 5188 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377938 5188 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377941 5188 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377944 5188 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377948 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377950 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377953 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377956 5188 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377958 5188 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377961 5188 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377964 5188 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377966 5188 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377969 5188 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377972 5188 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377974 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377977 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:53.394602 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377979 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377982 5188 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377985 5188 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377988 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377990 5188 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377993 5188 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377995 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.377999 5188 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378001 5188 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378004 5188 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378007 5188 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378009 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378012 5188 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378014 5188 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378017 5188 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378020 5188 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378022 5188 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378026 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378029 5188 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378031 5188 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:53.395304 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378036 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378039 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378042 5188 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378044 5188 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378047 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378050 5188 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378052 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378055 5188 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378057 5188 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378060 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378063 5188 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378066 5188 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378069 5188 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378072 5188 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378076 5188 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378080 5188 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378083 5188 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378087 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378090 5188 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:53.396237 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378093 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378096 5188 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378099 5188 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378102 5188 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378105 5188 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378108 5188 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378111 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378113 5188 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378116 5188 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378118 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378123 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378126 5188 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378129 5188 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378133 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378136 5188 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378140 5188 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378143 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378146 5188 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378150 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378152 5188 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:53.397077 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378155 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378158 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378160 5188 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378163 5188 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378165 5188 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378168 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378191 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378195 5188 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378199 5188 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378202 5188 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378207 5188 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378211 5188 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378215 5188 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378218 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378221 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378224 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378227 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378230 5188 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378232 5188 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:53.397942 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378235 5188 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378238 5188 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378241 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378247 5188 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378250 5188 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378252 5188 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.378257 5188 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.382983 5188 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.383008 5188 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.383031 5188 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.383039 5188 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383126 5188 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383135 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383140 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383146 5188 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383151 5188 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383156 5188 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383162 5188 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383167 5188 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383171 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:53.398795 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383176 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383180 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383184 5188 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383189 5188 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383193 5188 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383198 5188 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383203 5188 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383207 5188 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383213 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383217 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383221 5188 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383226 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383230 5188 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383234 5188 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383240 5188 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383245 5188 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383249 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383253 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383258 5188 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:53.399692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383262 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383268 5188 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383273 5188 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383277 5188 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383282 5188 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383286 5188 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383290 5188 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383296 5188 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383300 5188 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383304 5188 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383309 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383313 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383317 5188 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383339 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383343 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383347 5188 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383352 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383356 5188 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383361 5188 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383365 5188 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:53.400542 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383369 5188 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383374 5188 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383380 5188 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383384 5188 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383388 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383392 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383396 5188 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383401 5188 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383405 5188 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383410 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383414 5188 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383419 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383423 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383428 5188 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383434 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383438 5188 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383442 5188 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383447 5188 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383451 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:53.401437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383455 5188 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383460 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383464 5188 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383468 5188 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383475 5188 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383481 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383488 5188 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383504 5188 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383509 5188 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383514 5188 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383519 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383523 5188 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383528 5188 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383532 5188 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383537 5188 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383541 5188 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383546 5188 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383550 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383554 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:53.402252 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383558 5188 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383563 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383567 5188 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383571 5188 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383576 5188 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383580 5188 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383584 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383589 5188 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383593 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383599 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383603 5188 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383607 5188 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383612 5188 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383616 5188 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383620 5188 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383624 5188 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383629 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383633 5188 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383637 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:53.403097 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383641 5188 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.383686 5188 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383922 5188 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383931 5188 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383936 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383940 5188 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383945 5188 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383949 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383954 5188 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383958 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383963 5188 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383967 5188 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383972 5188 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383976 5188 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383980 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:53.403902 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383984 5188 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383991 5188 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.383998 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384005 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384009 5188 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384014 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384019 5188 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384023 5188 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384028 5188 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384033 5188 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384037 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384044 5188 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384051 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384057 5188 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384061 5188 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384065 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384071 5188 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384075 5188 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:53.404586 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384079 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384083 5188 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384087 5188 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384092 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384096 5188 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384101 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384105 5188 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384109 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384114 5188 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384118 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384122 5188 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384126 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384131 5188 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384135 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384140 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384144 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384148 5188 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384153 5188 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384157 5188 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384161 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:53.405653 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384165 5188 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384170 5188 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384174 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384178 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384183 5188 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384187 5188 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384192 5188 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384196 5188 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384200 5188 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384205 5188 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384209 5188 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384213 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384218 5188 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384222 5188 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384226 5188 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384230 5188 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384234 5188 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384239 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384243 5188 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384253 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:53.406607 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384258 5188 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384262 5188 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384266 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384271 5188 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384275 5188 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384280 5188 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384284 5188 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384288 5188 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384292 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384297 5188 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384301 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384305 5188 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384309 5188 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384313 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384337 5188 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384342 5188 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384348 5188 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384352 5188 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384357 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384361 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:53.407577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384365 5188 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384370 5188 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384374 5188 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384379 5188 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384384 5188 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384388 5188 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384393 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384397 5188 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384402 5188 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384406 5188 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384410 5188 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384417 5188 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384421 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384426 5188 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384430 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.384471 5188 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:53.408533 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384530 5188 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384538 5188 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384543 5188 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384548 5188 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384552 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384557 5188 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384563 5188 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384569 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384574 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384579 5188 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384583 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384589 5188 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384594 5188 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384600 5188 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384604 5188 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384608 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384615 5188 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384620 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:53.409372 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384626 5188 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384630 5188 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384634 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384639 5188 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384643 5188 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384648 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384652 5188 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384656 5188 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384661 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384666 5188 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384670 5188 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384675 5188 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384678 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384682 5188 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384686 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384690 5188 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384694 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384699 5188 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384703 5188 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:53.410178 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384709 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384713 5188 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384718 5188 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384722 5188 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384727 5188 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384731 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384737 5188 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384741 5188 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384746 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384751 5188 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384755 5188 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384760 5188 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384764 5188 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384769 5188 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384773 5188 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384777 5188 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384782 5188 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384786 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384790 5188 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384794 5188 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:53.411007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384798 5188 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384803 5188 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384807 5188 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384812 5188 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384816 5188 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384821 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384825 5188 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384829 5188 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384833 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384838 5188 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384842 5188 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384846 5188 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384852 5188 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384856 5188 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384861 5188 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384865 5188 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384869 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384873 5188 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384879 5188 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384884 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:53.411931 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384888 5188 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384893 5188 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384898 5188 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384902 5188 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384908 5188 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384912 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384917 5188 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384921 5188 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384925 5188 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384930 5188 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384934 5188 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384938 5188 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384943 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384948 5188 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384952 5188 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384956 5188 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384961 5188 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384965 5188 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384969 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:53.412777 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384974 5188 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384978 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384983 5188 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384989 5188 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384994 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.384998 5188 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.385003 5188 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.385007 5188 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.385011 5188 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:53.385015 5188 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.385101 5188 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.386811 5188 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.386900 5188 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.387187 5188 server.go:1014] "Starting client certificate rotation" Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.387336 5188 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.387395 5188 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 21:10:11 +0000 UTC" deadline="2028-06-08 02:42:15.781128727 +0000 UTC" Oct 31 21:15:53.413603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.387422 5188 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14045h26m22.393708098s" Oct 31 21:15:53.414302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.387961 5188 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:53.414302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.391083 5188 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:53.414302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.392295 5188 log.go:25] "Validated CRI v1 runtime API" Oct 31 21:15:53.414302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.394057 5188 log.go:25] "Validated CRI v1 image API" Oct 31 21:15:53.414302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.394820 5188 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 21:15:53.414302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.397882 5188 fs.go:135] Filesystem UUIDs: map[20a9b5ca-2a09-4067-b942-bda7fb73d8e9:/dev/nvme0n1p4 73b12867-1cf6-4b74-88e9-7f31b2a4582b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 31 21:15:53.414679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.397911 5188 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/037b736e04958f93ebd6345cab2da0d2b65d0a0d939e5c7283a74e18467e7c3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/037b736e04958f93ebd6345cab2da0d2b65d0a0d939e5c7283a74e18467e7c3a/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/078a12eed37966aef9fb378b99a4a30c60da7a6072fd39c38fab49c2d3824d04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/078a12eed37966aef9fb378b99a4a30c60da7a6072fd39c38fab49c2d3824d04/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15ec11997607027ae5066a249c611525380c2ac036280734b118e4ee59cc7862/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15ec11997607027ae5066a249c611525380c2ac036280734b118e4ee59cc7862/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16a63ca010aea713e9ad65816a00f6398257af06dac75fed21210be7279314b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16a63ca010aea713e9ad65816a00f6398257af06dac75fed21210be7279314b5/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e5b3d4df07a02a288eb471c3cf70a561cf4f36ebefb80f5ea0f14eee32e066d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e5b3d4df07a02a288eb471c3cf70a561cf4f36ebefb80f5ea0f14eee32e066d/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23342e2a936832b22f8ab613a00a5f38fff3fa37c4299db5160ef977d669bb0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23342e2a936832b22f8ab613a00a5f38fff3fa37c4299db5160ef977d669bb0c/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24ad4aaa02fcb3c760a3fe8813664d0215626be148def102ca59e19d7e3308f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24ad4aaa02fcb3c760a3fe8813664d0215626be148def102ca59e19d7e3308f5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/297bc8e6371cfa9f4f146cee5f2ec511763b3867bdd0ce42e7a9d610c7f5889d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/297bc8e6371cfa9f4f146cee5f2ec511763b3867bdd0ce42e7a9d610c7f5889d/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dd955e49284c880e75f26845637a762eb0802737a3ea7b18174e9f566869edf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dd955e49284c880e75f26845637a762eb0802737a3ea7b18174e9f566869edf/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/387844b07ce349c89d99b6c1db13f3c084e549bc3a56626339033905e54ffc94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/387844b07ce349c89d99b6c1db13f3c084e549bc3a56626339033905e54ffc94/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4397200e77b1c2a7f7a1a3354f9066a6a7193afa7d85f8ab9f5c97e84ebedf11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4397200e77b1c2a7f7a1a3354f9066a6a7193afa7d85f8ab9f5c97e84ebedf11/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c49701a585b3822378151f7d11449440bd81d2c770c1b0a892bc1c21ec85717/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c49701a585b3822378151f7d11449440bd81d2c770c1b0a892bc1c21ec85717/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66d4bcc4c3829affdf35b569dba49e789ff46376f4feaf1461e095084f0a3d9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66d4bcc4c3829affdf35b569dba49e789ff46376f4feaf1461e095084f0a3d9b/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67f935e7f23e653d91ae48b9757785164fe436dfcb3cfd1aa3ac377e5523a6f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67f935e7f23e653d91ae48b9757785164fe436dfcb3cfd1aa3ac377e5523a6f7/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b41925f038f7f1b008e0dc442614db444df41bcc0e657bcba4b7247f54cb1cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b41925f038f7f1b008e0dc442614db444df41bcc0e657bcba4b7247f54cb1cf/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cb8261c54eb4d093fb57b35866eeed8890e0e79f0aafa1f4ef72800cf166673/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cb8261c54eb4d093fb57b35866eeed8890e0e79f0aafa1f4ef72800cf166673/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cc24fa0fe204e03da7301b3ec40e6f21239820cd57a5cbc31c8ec56643306ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cc24fa0fe204e03da7301b3ec40e6f21239820cd57a5cbc31c8ec56643306ef/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c0d5c17583db4e70c67c6119379ee93e013f8fc26f49b2ec07eb16f785787f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c0d5c17583db4e70c67c6119379ee93e013f8fc26f49b2ec07eb16f785787f0/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b9c15d63056b516488b4f8bd97f8abfe7a9b2462a440559ec4553afe3dcdf6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b9c15d63056b516488b4f8bd97f8abfe7a9b2462a440559ec4553afe3dcdf6c/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/946256b8d18c3f7253e86f5402267c7e80521b1a1326694aab8d6727c85c52fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/946256b8d18c3f7253e86f5402267c7e80521b1a1326694aab8d6727c85c52fd/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a4733ea3a1a5f45ada86ffbae0898784ce87017407cf5e6d1b5cc7aa469b05c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a4733ea3a1a5f45ada86ffbae0898784ce87017407cf5e6d1b5cc7aa469b05c/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5b9999dedd757a42d09d4f770e53a67a8585c6399d7c6d68486664b6ddcfd9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5b9999dedd757a42d09d4f770e53a67a8585c6399d7c6d68486664b6ddcfd9a/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b46c20bb76e6bce8e4e792dc5ccb61793ff1e4d945aeda835f6f484a8123866e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b46c20bb76e6bce8e4e792dc5ccb61793ff1e4d945aeda835f6f484a8123866e/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c483d73912b3bed0811ff9aa7cc9ae3e170f7865023d4c8c843a20c766f7fec3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c483d73912b3bed0811ff9aa7cc9ae3e170f7865023d4c8c843a20c766f7fec3/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2849f1ebd647fe71e6d119c469c77e22cc3b8ce434a3832c1622ec52aaa26c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2849f1ebd647fe71e6d119c469c77e22cc3b8ce434a3832c1622ec52aaa26c7/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd425a165bedcbef0d9b6e7c11ab02987a2dc413dede5b11e287dd50ce339fff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd425a165bedcbef0d9b6e7c11ab02987a2dc413dede5b11e287dd50ce339fff/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b23775a-33a5-47b0-ac74-fb3862912542/volumes/kubernetes.io~projected/kube-api-access-7hljm:{mountpoint:/var/lib/kubelet/pods/1b23775a-33a5-47b0-ac74-fb3862912542/volumes/kubernetes.io~projected/kube-api-access-7hljm major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b23775a-33a5-47b0-ac74-fb3862912542/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1b23775a-33a5-47b0-ac74-fb3862912542/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21a28634-04b5-4d1b-bf14-6bdc3b894f2d/volumes/kubernetes.io~projected/kube-api-access-cw9ch:{mountpoint:/var/lib/kubelet/pods/21a28634-04b5-4d1b-bf14-6bdc3b894f2d/volumes/kubernetes.io~projected/kube-api-access-cw9ch major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21a28634-04b5-4d1b-bf14-6bdc3b894f2d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/21a28634-04b5-4d1b-bf14-6bdc3b894f2d/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5/volumes/kubernetes.io~projected/kube-api-access-p5f5z:{mountpoint:/var/lib/kubelet/pods/25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5/volumes/kubernetes.io~projected/kube-api-access-p5f5z major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29050b65-73ea-446c-98b6-76ce5438f0c9/volumes/kubernetes.io~projected/kube-api-access-7dnm6:{mountpoint:/var/lib/kubelet/pods/29050b65-73ea-446c-98b6-76ce5438f0c9/volumes/kubernetes.io~projected/kube-api-access-7dnm6 major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31682289-5cba-480d-88a4-c71e9cd6244b/volumes/kubernetes.io~projected/kube-api-access-lxlcr:{mountpoint:/var/lib/kubelet/pods/31682289-5cba-480d-88a4-c71e9cd6244b/volumes/kubernetes.io~projected/kube-api-access-lxlcr major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31682289-5cba-480d-88a4-c71e9cd6244b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/31682289-5cba-480d-88a4-c71e9cd6244b/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3616b3de-b6e6-497f-8546-8fef766c309f/volumes/kubernetes.io~projected/kube-api-access-f58tf:{mountpoint:/var/lib/kubelet/pods/3616b3de-b6e6-497f-8546-8fef766c309f/volumes/kubernetes.io~projected/kube-api-access-f58tf major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936/volumes/kubernetes.io~projected/kube-api-access-hpcxl:{mountpoint:/var/lib/kubelet/pods/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936/volumes/kubernetes.io~projected/kube-api-access-hpcxl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47354b4e-2a35-40be-a539-07dad787bd97/volumes/kubernetes.io~projected/kube-api-access-8jhwr:{mountpoint:/var/lib/kubelet/pods/47354b4e-2a35-40be-a539-07dad787bd97/volumes/kubernetes.io~projected/kube-api-access-8jhwr major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4873e537-31a8-4e6d-86bd-c54024d94c39/volumes/kubernetes.io~projected/kube-api-access-4rd6p:{mountpoint:/var/lib/kubelet/pods/4873e537-31a8-4e6d-86bd-c54024d94c39/volumes/kubernetes.io~projected/kube-api-access-4rd6p major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bf7148c-69be-45a9-bd56-7457a3929df8/volumes/kubernetes.io~projected/kube-api-access-ktw9c:{mountpoint:/var/lib/kubelet/pods/4bf7148c-69be-45a9-bd56-7457a3929df8/volumes/kubernetes.io~projected/kube-api-access-ktw9c major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6/volumes/kubernetes.io~projected/kube-api-access-bjcwz:{mountpoint:/var/lib/kubelet/pods/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6/volumes/kubernetes.io~projected/kube-api-access-bjcwz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/595edb9d-3370-41bd-82c3-af8df664ad90/volumes/kubernetes.io~projected/kube-api-access-j274g:{mountpoint:/var/lib/kubelet/pods/595edb9d-3370-41bd-82c3-af8df664ad90/volumes/kubernetes.io~projected/kube-api-access-j274g major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/595edb9d-3370-41bd-82c3-af8df664ad90/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/595edb9d-3370-41bd-82c3-af8df664ad90/volumes/kubernetes.io~secret/signing-key major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~projected/kube-api-access-j6qmg:{mountpoint:/var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~projected/kube-api-access-j6qmg major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/626d59af-b777-449a-87cc-23e763dd495f/volumes/kubernetes.io~projected/kube-api-access-725ms:{mountpoint:/var/lib/kubelet/pods/626d59af-b777-449a-87cc-23e763dd495f/volumes/kubernetes.io~projected/kube-api-access-725ms major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/672a2ce2-d45d-4f0c-a75a-b018105640aa/volumes/kubernetes.io~projected/kube-api-access-455ft:{mountpoint:/var/lib/kubelet/pods/672a2ce2-d45d-4f0c-a75a-b018105640aa/volumes/kubernetes.io~projected/kube-api-access-455ft major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6805be97-0a2e-490c-a50c-6700f15c4abc/volumes/kubernetes.io~projected/kube-api-access-vpsd4:{mountpoint:/var/lib/kubelet/pods/6805be97-0a2e-490c-a50c-6700f15c4abc/volumes/kubernetes.io~projected/kube-api-access-vpsd4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7245fb80-34d5-471b-984e-7fbfe2b1113c/volumes/kubernetes.io~projected/kube-api-access-whfjx:{mountpoint:/var/lib/kubelet/pods/7245fb80-34d5-471b-984e-7fbfe2b1113c/volumes/kubernetes.io~projected/kube-api-access-whfjx major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72751047-bc3c-4f2d-9655-e23d5211d346/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/72751047-bc3c-4f2d-9655-e23d5211d346/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ed572b5-9b0c-4924-af2f-87703ed1b5e6/volumes/kubernetes.io~projected/kube-api-access-98zhq:{mountpoint:/var/lib/kubelet/pods/7ed572b5-9b0c-4924-af2f-87703ed1b5e6/volumes/kubernetes.io~projected/kube-api-access-98zhq major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/volumes/kubernetes.io~projected/kube-api-access-bxpcl:{mountpoint:/var/lib/kubelet/pods/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/volumes/kubernetes.io~projected/kube-api-access-bxpcl major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f8d2547-a263-47ff-aaa9-780ea67176a5/volumes/kubernetes.io~projected/kube-api-access-gs5kk:{mountpoint:/var/lib/kubelet/pods/9f8d2547-a263-47ff-aaa9-780ea67176a5/volumes/kubernetes.io~projected/kube-api-access-gs5kk major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4772ac2-3076-4099-89da-cb3be3d4387c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a4772ac2-3076-4099-89da-cb3be3d4387c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4772ac2-3076-4099-89da-cb3be3d4387c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a4772ac2-3076-4099-89da-cb3be3d4387c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4772ac2-3076-4099-89da-cb3be3d4387c/volumes/kubernetes.io~projected/kube-api-access-tfgnv:{mountpoint:/var/lib/kubelet/pods/a4772ac2-3076-4099-89da-cb3be3d4387c/volumes/kubernetes.io~projected/kube-api-access-tfgnv major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a/volumes/kubernetes.io~projected/kube-api-access-t57pd:{mountpoint:/var/lib/kubelet/pods/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a/volumes/kubernetes.io~projected/kube-api-access-t57pd major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac6429fb-48a7-4691-9079-4ed168f1da32/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/ac6429fb-48a7-4691-9079-4ed168f1da32/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/add13542-315e-4dde-a5fa-728aaf6fdc6c/volumes/kubernetes.io~projected/kube-api-access-mwtqf:{mountpoint:/var/lib/kubelet/pods/add13542-315e-4dde-a5fa-728aaf6fdc6c/volumes/kubernetes.io~projected/kube-api-access-mwtqf major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/add13542-315e-4dde-a5fa-728aaf6fdc6c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/add13542-315e-4dde-a5fa-728aaf6fdc6c/volumes/kubernetes.io~secret/cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6f98741-1d48-4124-9acb-8c0bf31dbe60/volumes/kubernetes.io~projected/kube-api-access-pcnhc:{mountpoint:/var/lib/kubelet/pods/b6f98741-1d48-4124-9acb-8c0bf31dbe60/volumes/kubernetes.io~projected/kube-api-access-pcnhc major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c01f3957-4a49-41ba-812b-ea2e74d3a7a8/volumes/kubernetes.io~projected/kube-api-access-t7fwg:{mountpoint:/var/lib/kubelet/pods/c01f3957-4a49-41ba-812b-ea2e74d3a7a8/volumes/kubernetes.io~projected/kube-api-access-t7fwg major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c929e38e-147c-41f9-ad34-b0af48044e09/volumes/kubernetes.io~projected/kube-api-access-fc8sb:{mountpoint:/var/lib/kubelet/pods/c929e38e-147c-41f9-ad34-b0af48044e09/volumes/kubernetes.io~projected/kube-api-access-fc8sb major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~projected/kube-api-access-k5vtd:{mountpoint:/var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~projected/kube-api-access-k5vtd major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2a47228-9b40-4833-bb84-970f4782e67b/volumes/kubernetes.io~projected/kube-api-access-b445d:{mountpoint:/var/lib/kubelet/pods/d2a47228-9b40-4833-bb84-970f4782e67b/volumes/kubernetes.io~projected/kube-api-access-b445d major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2a47228-9b40-4833-bb84-970f4782e67b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d2a47228-9b40-4833-bb84-970f4782e67b/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e49a2d50-dc82-4983-9425-053dfe903ddd/volumes/kubernetes.io~projected/kube-api-access-4rxlh:{mountpoint:/var/lib/kubelet/pods/e49a2d50-dc82-4983-9425-053dfe903ddd/volumes/kubernetes.io~projected/kube-api-access-4rxlh major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6924bea-83c3-443f-a95d-581edf9aaf7b/volumes/kubernetes.io~projected/kube-api-access-mjqft:{mountpoint:/var/lib/kubelet/pods/e6924bea-83c3-443f-a95d-581edf9aaf7b/volumes/kubernetes.io~projected/kube-api-access-mjqft major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7d03911-7706-47b8-928e-6ffea21fdb26/volumes/kubernetes.io~projected/kube-api-access-927v6:{mountpoint:/var/lib/kubelet/pods/f7d03911-7706-47b8-928e-6ffea21fdb26/volumes/kubernetes.io~projected/kube-api-access-927v6 major:0 minor:223 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c93207130838a37eeae5d1260192b27e115549caa47424e02b999926b0969a3b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9f9f6d50c3014de8196ef2df29996575222f15df0a6cb269c518f3667f1820ea/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/df3c5625113c90e620072a1432088bd16a322e7cc3c2cf3f5c328a6911ed700b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0e1e6752c31b185685a5ef78f1f042c81f679ffd69821d2682e366d386fda822/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ec7fc4c12c4c172e8c138e2e53d5bee794613bce1173040418fe6dfde2e78193/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/165ddd429c87a87d5aa9c065461e7beff9a8709b1800728a4fa9ce69f4701f4e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/15db867956d40260997363cd7a23675cf0b06fabaeb46a13ee8c9c8b1b3f72bf/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/13b0aea2c6e867428df3eaab9f24cd8c87dd7cc3058bc5647e4770c888cb8042/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/5c48d2add642d50698fdb6f0d97e8291f85743b639cd2dcb33b9920158409216/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/df958bc299f5e50915db4ddc75cbfadf9c9f2b2927eb2edbe23599719ab387b8/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e82a18c90ae68dc37e62299e15689d3e8046580fbeb852d5ac9480cc1a5131f1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/0eeac6276bcfaa2e66f113cdb141f48bde06eed7877e4b5607a81d05ad2bec68/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/02d1e6546d36162668d7e6317c692b6a9b33e6d7713ea6f5c2ab24bfe160f8e1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/26ca6b4a41fa45ffe161ab5366784352e80bc02123c090fbe26c521cdb0648d9/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4dc53a215a7d5f573236e975773d1dcaa885dde5f080ccc434c97b19b80117b5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/757082c1ebf39d82d99fddebd77464714b7328cc1748b4f745509349ac57ef36/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/221ad004964484bf34ed7b85daf42bd0bab0e108faae2bb8a8ca67493856d45e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/759694dcaa61c6a42cb710706857f7785c71cdf76a1b21669dfa88ddc286914b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/containers/storage/overlay/8f758e155500ab5ef5c00f010cd145b8983c1dfeaf0311cf49b5412f67ffaf38/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/32677d8bf95645ed9da01ad7ea81ca84a7fb06fab30ffb8f6582681eb94bd73a/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9f1e49575870ca245416f3186eba420f218bf5c1f266ca277f247ae2c878e194/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9e5c04a8a4482c2d41d4a1a8645b6cfe025fb48bd50ceeb7553d92b93f37d22a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b758b0172e23ab3fac1bcee24598fcfd23019eee309bfdfe1fe2039361b21a9e/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f304ef8559d4597086b372c0e2c61749ea153b9f7b828b86dfe516115ebaf454/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/2e4f2a2db5686372e6b3af540ee8c3f25cab7985c41fa0cd6a9f3ffc641fa332/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f3a58d17b093a5a6661d3e72815eaea15ceed5dc7317b00acacfb8d756d89393/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/9cb43b7cdadaac17490df08bf71a7830683a5743b3891ecac58f9583860498c0/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/007e938e9e55c4432502778ff882d3a3964453c2a91476c98a7f05052927d640/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/dfcd9e77e668a206d0a82edd236adb63b0f69a02c488e8fd99eaf89bf4f13bc2/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/9b1bca669ee538318ea97c8d82b5eda8e09479082362faacbfc9fa331a6c2fd1/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/b4c05f76dc5aaac162f573b5f168666333dc37b60334c308310f9c81368a85e6/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/9095ca81d015627ae85a45af73c2a505fdd5fcd7af16fa5e747eff9723005515/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/f7e47b0cf3db464fda3c67bd18dcf33eedbc4f6c8ccdf9c46ba51cae5dc6dce6/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/bd23db501ee2d7d7f2602ccf5a38ac3f54408e10a093d80f49e33d872af4ce02/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/0a4adf198cbea8aa55485009d2896b32dcac32dc9f9f010a36e8474e72666555/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/cf40ac333ccc2a979df8bdc2c7ff89dc2a2a30e499840541a85fc260d148db4f/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/d8e7171d11d5ee6cdcfe04c52990ed7f41523964875740b0018000e06e41f0e6/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/be0e95c895fe694e37021683aa2f42ccc5a571c6f1480d3a8e22d9c1c8fc8baa/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/93a1b6973843ec9d5a80d1b1c16848ee9e98318b1e3b1f0343a3ed8042c8d004/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/c27c9e20c322f4442c152562f3d18caf3667e37ae367cff4b85af9e54ae4a792/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/eaf2da2e88ee4b8e592661e85ba2407772c48aaf0a65c026d6c5e4e43558649e/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/9ac29cfed9513582359cfb07714311f0fb880eafa2a90f1e9c86384c9e7601d0/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/015ce9f1c143fc609a41f7299c5144bde4acc55c8f37ef1bf304263ef0d9d1c4/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/a45fd0dc018ac450a3c100af83267bf33a54c5d64d619ce7258dfb84c51d4be4/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/8a5de9a304de7a16495aed907a4d04b4bc864f6072e2b8a514551337554dc9ae/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/d6ee93bddecc5ae65ad7f9f21c6920f58b79caef80498d0992f110f2ea7ea2f0/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/89a69d7726febff6bdab46ba736d2af03ffe631034dbe9cf65c7b88ae02af90e/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/6f3f922512cd6cbdf9262be89b43a183219ad195b0cb5ed691d092e99c599826/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/cb4b0c64b0d71b4b6cf732bc8f508e047f4cabeefe0a9d2a61d2a8344b057801/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/58c220fe38ddcb0b663b606e1895d655b2fddd0e0e3d9607f593d4384267719e/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/eae97c3929c592130af68acd73da0fa04deeac266a31520902cb5f653afc7e3a/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/d45f9cc4e93a51b775ae184e730902f170c8657e39350f9f75b634bd7ad912a7/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b243f90bc63ad2449522f9d8489c8ca9a33d7e45dc4de2fa4711ae3dfc4dfa70/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0f452726a425b1a9b79990454559358da8e51ea204b9f50148856202a9d87566/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/e914eb0a6e0ae266aa0bff5e884bb39d3452150238434b6cc49a69b0bd8b2e19/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/332844330e7f347dd39902075dadf19c607097cc38b99316be873546c8a3370d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/26c73ef89873bd61b0d81eadf21f93e514fb1d393db2a80f6efe49e10e88f345/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9f0132a950de0504d6bab7893e24f7d74e03562b801ac9ad2b6879c92bd0189d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/9e3a5fdecd6698dd9d9c1d649ec33c91b0d4cef8bffe075688733932189381d1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e69a875569faa83c3ac189d36fe055f60a78aa2171b4f32c7443fb2ef7e37cf3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c184f927f21fff4ab3cfe483291f0ddea9a32aebb2514f5d57a132d8bb704ae9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/ac7dc09c73c006f52e052a1d9ee59548d431a648bef201ee67abbdb0f75e39b7/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.410616 5188 manager.go:217] Machine: {Timestamp:2026-10-31 21:15:53.409775602 +0000 UTC m=+0.102732991 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099149 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fe8ecc0817846daff2017c7427327 SystemUUID:ec2fe8ec-c081-7846-daff-2017c7427327 BootID:96a4953a-69d6-40a0-8e26-08435db256a4 Filesystems:[{Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31682289-5cba-480d-88a4-c71e9cd6244b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9f8d2547-a263-47ff-aaa9-780ea67176a5/volumes/kubernetes.io~projected/kube-api-access-gs5kk DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b46c20bb76e6bce8e4e792dc5ccb61793ff1e4d945aeda835f6f484a8123866e/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b23775a-33a5-47b0-ac74-fb3862912542/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b6f98741-1d48-4124-9acb-8c0bf31dbe60/volumes/kubernetes.io~projected/kube-api-access-pcnhc DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/626d59af-b777-449a-87cc-23e763dd495f/volumes/kubernetes.io~projected/kube-api-access-725ms DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~projected/kube-api-access-j6qmg DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6924bea-83c3-443f-a95d-581edf9aaf7b/volumes/kubernetes.io~projected/kube-api-access-mjqft DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4772ac2-3076-4099-89da-cb3be3d4387c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dd955e49284c880e75f26845637a762eb0802737a3ea7b18174e9f566869edf/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24ad4aaa02fcb3c760a3fe8813664d0215626be148def102ca59e19d7e3308f5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ac6429fb-48a7-4691-9079-4ed168f1da32/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/078a12eed37966aef9fb378b99a4a30c60da7a6072fd39c38fab49c2d3824d04/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2a47228-9b40-4833-bb84-970f4782e67b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d2a47228-9b40-4833-bb84-970f4782e67b/volumes/kubernetes.io~projected/kube-api-access-b445d DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2849f1ebd647fe71e6d119c469c77e22cc3b8ce434a3832c1622ec52aaa26c7/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/29050b65-73ea-446c-98b6-76ce5438f0c9/volumes/kubernetes.io~projected/kube-api-access-7dnm6 DeviceMajor:0 DeviceMinor:324 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4873e537-31a8-4e6d-86bd-c54024d94c39/volumes/kubernetes.io~projected/kube-api-access-4rd6p DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b9c15d63056b516488b4f8bd97f8abfe7a9b2462a440559ec4553afe3dcdf6c/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/037b736e04958f93ebd6345cab2da0d2b65d0a0d939e5c7283a74e18467e7c3a/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7245fb80-34d5-471b-984e-7fbfe2b1113c/volumes/kubernetes.io~projected/kube-api-access-whfjx DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/595edb9d-3370-41bd-82c3-af8df664ad90/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/387844b07ce349c89d99b6c1db13f3c084e549bc3a56626339033905e54ffc94/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-178 DeviceMajor:0 DeviceMinor:178 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c01f3957-4a49-41ba-812b-ea2e74d3a7a8/volumes/kubernetes.io~projected/kube-api-access-t7fwg DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e49a2d50-dc82-4983-9425-053dfe903ddd/volumes/kubernetes.io~projected/kube-api-access-4rxlh DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/946256b8d18c3f7253e86f5402267c7e80521b1a1326694aab8d6727c85c52fd/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c929e38e-147c-41f9-ad34-b0af48044e09/volumes/kubernetes.io~projected/kube-api-access-fc8sb DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a4733ea3a1a5f45ada86ffbae0898784ce87017407cf5e6d1b5cc7aa469b05c/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a/volumes/kubernetes.io~projected/kube-api-access-t57pd DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72751047-bc3c-4f2d-9655-e23d5211d346/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~projected/kube-api-access-k5vtd DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66d4bcc4c3829affdf35b569dba49e789ff46376f4feaf1461e095084f0a3d9b/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b41925f038f7f1b008e0dc442614db444df41bcc0e657bcba4b7247f54cb1cf/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/volumes/kubernetes.io~projected/kube-api-access-bxpcl DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7ed572b5-9b0c-4924-af2f-87703ed1b5e6/volumes/kubernetes.io~projected/kube-api-access-98zhq DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/add13542-315e-4dde-a5fa-728aaf6fdc6c/volumes/kubernetes.io~projected/kube-api-access-mwtqf DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5b9999dedd757a42d09d4f770e53a67a8585c6399d7c6d68486664b6ddcfd9a/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e5b3d4df07a02a288eb471c3cf70a561cf4f36ebefb80f5ea0f14eee32e066d/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21a28634-04b5-4d1b-bf14-6bdc3b894f2d/volumes/kubernetes.io~projected/kube-api-access-cw9ch DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23342e2a936832b22f8ab613a00a5f38fff3fa37c4299db5160ef977d669bb0c/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67f935e7f23e653d91ae48b9757785164fe436dfcb3cfd1aa3ac377e5523a6f7/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/add13542-315e-4dde-a5fa-728aaf6fdc6c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5/volumes/kubernetes.io~projected/kube-api-access-p5f5z DeviceMajor:0 DeviceMinor:335 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/297bc8e6371cfa9f4f146cee5f2ec511763b3867bdd0ce42e7a9d610c7f5889d/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c49701a585b3822378151f7d11449440bd81d2c770c1b0a892bc1c21ec85717/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c0d5c17583db4e70c67c6119379ee93e013f8fc26f49b2ec07eb16f785787f0/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47354b4e-2a35-40be-a539-07dad787bd97/volumes/kubernetes.io~projected/kube-api-access-8jhwr DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cb8261c54eb4d093fb57b35866eeed8890e0e79f0aafa1f4ef72800cf166673/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/31682289-5cba-480d-88a4-c71e9cd6244b/volumes/kubernetes.io~projected/kube-api-access-lxlcr DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15ec11997607027ae5066a249c611525380c2ac036280734b118e4ee59cc7862/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3616b3de-b6e6-497f-8546-8fef766c309f/volumes/kubernetes.io~projected/kube-api-access-f58tf DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd425a165bedcbef0d9b6e7c11ab02987a2dc413dede5b11e287dd50ce339fff/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936/volumes/kubernetes.io~projected/kube-api-access-hpcxl DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c483d73912b3bed0811ff9aa7cc9ae3e170f7865023d4c8c843a20c766f7fec3/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/595edb9d-3370-41bd-82c3-af8df664ad90/volumes/kubernetes.io~projected/kube-api-access-j274g DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6/volumes/kubernetes.io~projected/kube-api-access-bjcwz DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1b23775a-33a5-47b0-ac74-fb3862912542/volumes/kubernetes.io~projected/kube-api-access-7hljm DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/672a2ce2-d45d-4f0c-a75a-b018105640aa/volumes/kubernetes.io~projected/kube-api-access-455ft DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6805be97-0a2e-490c-a50c-6700f15c4abc/volumes/kubernetes.io~projected/kube-api-access-vpsd4 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21a28634-04b5-4d1b-bf14-6bdc3b894f2d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a4772ac2-3076-4099-89da-cb3be3d4387c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4772ac2-3076-4099-89da-cb3be3d4387c/volumes/kubernetes.io~projected/kube-api-access-tfgnv DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bf7148c-69be-45a9-bd56-7457a3929df8/volumes/kubernetes.io~projected/kube-api-access-ktw9c DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f7d03911-7706-47b8-928e-6ffea21fdb26/volumes/kubernetes.io~projected/kube-api-access-927v6 DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cc24fa0fe204e03da7301b3ec40e6f21239820cd57a5cbc31c8ec56643306ef/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4397200e77b1c2a7f7a1a3354f9066a6a7193afa7d85f8ab9f5c97e84ebedf11/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16a63ca010aea713e9ad65816a00f6398257af06dac75fed21210be7279314b5/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:037b736e04958f9 MacAddress:fa:39:d2:3e:d8:b5 Speed:10000 Mtu:8901} {Name:078a12eed37966a MacAddress:6e:7f:90:1b:05:32 Speed:10000 Mtu:8901} {Name:15ec11997607027 MacAddress:f6:00:e7:68:22:41 Speed:10000 Mtu:8901} {Name:16a63ca010aea71 MacAddress:7e:f3:7f:83:99:db Speed:10000 Mtu:8901} {Name:23342e2a936832b MacAddress:ba:f1:03:9f:74:66 Speed:10000 Mtu:8901} {Name:297bc8e6371cfa9 MacAddress:16:81:5d:2c:fe:c1 Speed:10000 Mtu:8901} {Name:2dd955e49284c88 MacAddress:3a:69:8a:81:4a:2b Speed:10000 Mtu:8901} {Name:5c49701a585b382 MacAddress:26:85:39:d4:83:4a Speed:10000 Mtu:8901} {Name:66d4bcc4c3829af MacAddress:1a:99:b6:a9:e2:c4 Speed:10000 Mtu:8901} {Name:6cc24fa0fe204e0 MacAddress:36:a4:7d:6b:66:4e Speed:10000 Mtu:8901} {Name:7c0d5c17583db4e MacAddress:6a:17:b4:43:22:93 Speed:10000 Mtu:8901} {Name:946256b8d18c3f7 MacAddress:32:a2:dd:6d:65:5f Speed:10000 Mtu:8901} {Name:9a4733ea3a1a5f4 MacAddress:3e:97:15:5f:ed:0c Speed:10000 Mtu:8901} {Name:a5b9999dedd757a MacAddress:9e:8a:45:20:14:14 Speed:10000 Mtu:8901} {Name:b46c20bb76e6bce MacAddress:c6:fd:d7:f1:cc:6a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f8:9a:8a:2d Speed:0 Mtu:9001} {Name:br-int MacAddress:7a:9d:f5:3c:16:e0 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f8:9a:8a:2d Speed:0 Mtu:9001} {Name:fd425a165bedcbe MacAddress:3e:49:ef:07:16:e0 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fe:3e:16:a5:53:2f Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7e:7d:14:e6:be:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.411288 5188 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.411375 5188 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.411779 5188 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.411815 5188 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-21.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412043 5188 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412056 5188 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412075 5188 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412103 5188 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412131 5188 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412373 5188 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412522 5188 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412580 5188 kubelet.go:493] "Attempting to sync node with API server" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412596 5188 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412614 5188 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412628 5188 kubelet.go:405] "Adding apiserver pod source" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.412640 5188 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.414226 5188 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.414241 5188 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 21:15:53.414983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.414984 5188 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 21:15:53.419909 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.419879 5188 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 21:15:53.419909 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.419906 5188 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 21:15:53.420076 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420061 5188 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 21:15:53.420402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420386 5188 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 21:15:53.420443 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420433 5188 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 21:15:53.420564 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420555 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 21:15:53.420598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420570 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 21:15:53.420598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420577 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 21:15:53.420598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420584 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 21:15:53.420598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420592 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 21:15:53.420704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420602 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 21:15:53.420704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420611 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 21:15:53.420704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420617 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 21:15:53.420704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420624 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 21:15:53.420704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420630 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 21:15:53.420704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420640 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 21:15:53.420704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420649 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 21:15:53.420704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420679 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 21:15:53.420951 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.420690 5188 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 21:15:53.424474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.424459 5188 server.go:1263] "Started kubelet" Oct 31 21:15:53.424585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.424553 5188 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 21:15:53.424641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.424566 5188 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 21:15:53.424694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.424658 5188 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 21:15:53.425195 ip-10-0-131-21 systemd[1]: Started Kubernetes Kubelet. Oct 31 21:15:53.425373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.425219 5188 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 21:15:53.426005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.425988 5188 server.go:317] "Adding debug handlers to kubelet server" Oct 31 21:15:53.428572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.428550 5188 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 21:15:53.428690 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.428603 5188 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 21:15:53.428774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.428652 5188 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 21:10:13 +0000 UTC" deadline="2028-04-15 20:40:17.090112798 +0000 UTC" Oct 31 21:15:53.428774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.428732 5188 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12767h24m23.661384089s" Oct 31 21:15:53.428880 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:53.428779 5188 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-21.ec2.internal\" not found" Oct 31 21:15:53.428880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.428822 5188 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 21:15:53.428880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.428848 5188 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 21:15:53.429154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.429138 5188 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 21:15:53.431001 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.430981 5188 factory.go:55] Registering systemd factory Oct 31 21:15:53.431146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.431006 5188 factory.go:223] Registration of the systemd container factory successfully Oct 31 21:15:53.431760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.431308 5188 factory.go:153] Registering CRI-O factory Oct 31 21:15:53.431760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.431352 5188 factory.go:223] Registration of the crio container factory successfully Oct 31 21:15:53.431760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.431420 5188 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 21:15:53.431760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.431446 5188 factory.go:103] Registering Raw factory Oct 31 21:15:53.431760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.431510 5188 manager.go:1199] Started watching for new ooms in manager Oct 31 21:15:53.432119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.432066 5188 manager.go:319] Starting recovery of all containers Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.436822 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43bf6948-d5fa-4e37-8f85-d2d9dd9d4936" volumeName="kubernetes.io/projected/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-kube-api-access-hpcxl" seLinuxMountContext="" Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.436973 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="595edb9d-3370-41bd-82c3-af8df664ad90" volumeName="kubernetes.io/projected/595edb9d-3370-41bd-82c3-af8df664ad90-kube-api-access-j274g" seLinuxMountContext="" Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.436997 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" volumeName="kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates" seLinuxMountContext="" Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.437011 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" volumeName="kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg" seLinuxMountContext="" Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.437095 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce941b3a-da84-4b86-944c-89f9f464a27f" volumeName="kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets" seLinuxMountContext="" Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.437117 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6" volumeName="kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-config" seLinuxMountContext="" Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.437131 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bd131db-e6d0-46c6-9a4b-85d3b541a9ec" volumeName="kubernetes.io/configmap/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-nginx-conf" seLinuxMountContext="" Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.437148 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b23775a-33a5-47b0-ac74-fb3862912542" volumeName="kubernetes.io/secret/1b23775a-33a5-47b0-ac74-fb3862912542-serving-cert" seLinuxMountContext="" Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.437167 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a28634-04b5-4d1b-bf14-6bdc3b894f2d" volumeName="kubernetes.io/configmap/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-config" seLinuxMountContext="" Oct 31 21:15:53.437276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.437241 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47354b4e-2a35-40be-a539-07dad787bd97" volumeName="kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-multus-daemon-config" seLinuxMountContext="" Oct 31 21:15:53.441396 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.437298 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="595edb9d-3370-41bd-82c3-af8df664ad90" volumeName="kubernetes.io/secret/595edb9d-3370-41bd-82c3-af8df664ad90-signing-key" seLinuxMountContext="" Oct 31 21:15:53.441495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441415 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce941b3a-da84-4b86-944c-89f9f464a27f" volumeName="kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates" seLinuxMountContext="" Oct 31 21:15:53.441495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441451 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce941b3a-da84-4b86-944c-89f9f464a27f" volumeName="kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca" seLinuxMountContext="" Oct 31 21:15:53.441495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441467 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="934a6c09-d6c4-4ae2-afbf-3bc19cc948c8" volumeName="kubernetes.io/projected/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-kube-api-access-bxpcl" seLinuxMountContext="" Oct 31 21:15:53.441495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441485 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4772ac2-3076-4099-89da-cb3be3d4387c" volumeName="kubernetes.io/projected/a4772ac2-3076-4099-89da-cb3be3d4387c-kube-api-access-tfgnv" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441499 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" volumeName="kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441511 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6805be97-0a2e-490c-a50c-6700f15c4abc" volumeName="kubernetes.io/projected/6805be97-0a2e-490c-a50c-6700f15c4abc-kube-api-access-vpsd4" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441526 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7245fb80-34d5-471b-984e-7fbfe2b1113c" volumeName="kubernetes.io/empty-dir/7245fb80-34d5-471b-984e-7fbfe2b1113c-data-volume" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441537 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" volumeName="kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-binary-copy" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441548 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce941b3a-da84-4b86-944c-89f9f464a27f" volumeName="kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441559 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" volumeName="kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441573 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac6429fb-48a7-4691-9079-4ed168f1da32" volumeName="kubernetes.io/secret/ac6429fb-48a7-4691-9079-4ed168f1da32-agent-certs" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441605 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c01f3957-4a49-41ba-812b-ea2e74d3a7a8" volumeName="kubernetes.io/projected/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-kube-api-access-t7fwg" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441619 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bf7148c-69be-45a9-bd56-7457a3929df8" volumeName="kubernetes.io/projected/4bf7148c-69be-45a9-bd56-7457a3929df8-kube-api-access-ktw9c" seLinuxMountContext="" Oct 31 21:15:53.441639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441632 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6" volumeName="kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-env-overrides" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441651 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7245fb80-34d5-471b-984e-7fbfe2b1113c" volumeName="kubernetes.io/projected/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-api-access-whfjx" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441664 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="934a6c09-d6c4-4ae2-afbf-3bc19cc948c8" volumeName="kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-default-certificate" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441675 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="934a6c09-d6c4-4ae2-afbf-3bc19cc948c8" volumeName="kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-stats-auth" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441700 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" volumeName="kubernetes.io/projected/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-kube-api-access-t57pd" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441711 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add13542-315e-4dde-a5fa-728aaf6fdc6c" volumeName="kubernetes.io/projected/add13542-315e-4dde-a5fa-728aaf6fdc6c-kube-api-access-mwtqf" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441722 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47354b4e-2a35-40be-a539-07dad787bd97" volumeName="kubernetes.io/projected/47354b4e-2a35-40be-a539-07dad787bd97-kube-api-access-8jhwr" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441740 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29050b65-73ea-446c-98b6-76ce5438f0c9" volumeName="kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441750 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" volumeName="kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441761 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29050b65-73ea-446c-98b6-76ce5438f0c9" volumeName="kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441780 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="672a2ce2-d45d-4f0c-a75a-b018105640aa" volumeName="kubernetes.io/projected/672a2ce2-d45d-4f0c-a75a-b018105640aa-kube-api-access-455ft" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441791 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6805be97-0a2e-490c-a50c-6700f15c4abc" volumeName="kubernetes.io/empty-dir/6805be97-0a2e-490c-a50c-6700f15c4abc-tmp-dir" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441801 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4772ac2-3076-4099-89da-cb3be3d4387c" volumeName="kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-tuned" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441828 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6f98741-1d48-4124-9acb-8c0bf31dbe60" volumeName="kubernetes.io/empty-dir/b6f98741-1d48-4124-9acb-8c0bf31dbe60-tmp-dir" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441840 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6f98741-1d48-4124-9acb-8c0bf31dbe60" volumeName="kubernetes.io/projected/b6f98741-1d48-4124-9acb-8c0bf31dbe60-kube-api-access-pcnhc" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441860 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce941b3a-da84-4b86-944c-89f9f464a27f" volumeName="kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441879 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2a47228-9b40-4833-bb84-970f4782e67b" volumeName="kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-tmp" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441896 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" volumeName="kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441908 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" volumeName="kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441929 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f8d2547-a263-47ff-aaa9-780ea67176a5" volumeName="kubernetes.io/projected/9f8d2547-a263-47ff-aaa9-780ea67176a5-kube-api-access-gs5kk" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441943 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2a47228-9b40-4833-bb84-970f4782e67b" volumeName="kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-service-ca-bundle" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441955 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2a47228-9b40-4833-bb84-970f4782e67b" volumeName="kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-snapshots" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441977 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2a47228-9b40-4833-bb84-970f4782e67b" volumeName="kubernetes.io/projected/d2a47228-9b40-4833-bb84-970f4782e67b-kube-api-access-b445d" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.441998 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6" volumeName="kubernetes.io/projected/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-kube-api-access-bjcwz" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442013 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b23775a-33a5-47b0-ac74-fb3862912542" volumeName="kubernetes.io/projected/1b23775a-33a5-47b0-ac74-fb3862912542-kube-api-access-7hljm" seLinuxMountContext="" Oct 31 21:15:53.442038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442035 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72751047-bc3c-4f2d-9655-e23d5211d346" volumeName="kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442059 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" volumeName="kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442073 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29050b65-73ea-446c-98b6-76ce5438f0c9" volumeName="kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442095 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43bf6948-d5fa-4e37-8f85-d2d9dd9d4936" volumeName="kubernetes.io/configmap/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-iptables-alerter-script" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442108 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47354b4e-2a35-40be-a539-07dad787bd97" volumeName="kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-cni-binary-copy" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442120 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="626d59af-b777-449a-87cc-23e763dd495f" volumeName="kubernetes.io/projected/626d59af-b777-449a-87cc-23e763dd495f-kube-api-access-725ms" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442142 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c01f3957-4a49-41ba-812b-ea2e74d3a7a8" volumeName="kubernetes.io/configmap/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-telemetry-config" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442166 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49a2d50-dc82-4983-9425-053dfe903ddd" volumeName="kubernetes.io/projected/e49a2d50-dc82-4983-9425-053dfe903ddd-kube-api-access-4rxlh" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442180 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b23775a-33a5-47b0-ac74-fb3862912542" volumeName="kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-config" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442202 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a28634-04b5-4d1b-bf14-6bdc3b894f2d" volumeName="kubernetes.io/projected/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-kube-api-access-cw9ch" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442216 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31682289-5cba-480d-88a4-c71e9cd6244b" volumeName="kubernetes.io/projected/31682289-5cba-480d-88a4-c71e9cd6244b-kube-api-access-lxlcr" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442229 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="add13542-315e-4dde-a5fa-728aaf6fdc6c" volumeName="kubernetes.io/secret/add13542-315e-4dde-a5fa-728aaf6fdc6c-cert" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442250 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce941b3a-da84-4b86-944c-89f9f464a27f" volumeName="kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442265 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2a47228-9b40-4833-bb84-970f4782e67b" volumeName="kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-trusted-ca-bundle" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442278 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b23775a-33a5-47b0-ac74-fb3862912542" volumeName="kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-trusted-ca" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442301 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c929e38e-147c-41f9-ad34-b0af48044e09" volumeName="kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442314 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7d03911-7706-47b8-928e-6ffea21fdb26" volumeName="kubernetes.io/projected/f7d03911-7706-47b8-928e-6ffea21fdb26-kube-api-access-927v6" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442341 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5" volumeName="kubernetes.io/projected/25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5-kube-api-access-p5f5z" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442365 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="595edb9d-3370-41bd-82c3-af8df664ad90" volumeName="kubernetes.io/configmap/595edb9d-3370-41bd-82c3-af8df664ad90-signing-cabundle" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442378 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" volumeName="kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442391 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="672a2ce2-d45d-4f0c-a75a-b018105640aa" volumeName="kubernetes.io/configmap/672a2ce2-d45d-4f0c-a75a-b018105640aa-serviceca" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442413 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6" volumeName="kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-script-lib" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442428 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4772ac2-3076-4099-89da-cb3be3d4387c" volumeName="kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-tmp" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442439 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2a47228-9b40-4833-bb84-970f4782e67b" volumeName="kubernetes.io/secret/d2a47228-9b40-4833-bb84-970f4782e67b-serving-cert" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442461 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21a28634-04b5-4d1b-bf14-6bdc3b894f2d" volumeName="kubernetes.io/secret/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-serving-cert" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442477 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31682289-5cba-480d-88a4-c71e9cd6244b" volumeName="kubernetes.io/configmap/31682289-5cba-480d-88a4-c71e9cd6244b-config" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442490 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31682289-5cba-480d-88a4-c71e9cd6244b" volumeName="kubernetes.io/secret/31682289-5cba-480d-88a4-c71e9cd6244b-serving-cert" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442512 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3616b3de-b6e6-497f-8546-8fef766c309f" volumeName="kubernetes.io/projected/3616b3de-b6e6-497f-8546-8fef766c309f-kube-api-access-f58tf" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442528 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac6429fb-48a7-4691-9079-4ed168f1da32" volumeName="kubernetes.io/configmap/ac6429fb-48a7-4691-9079-4ed168f1da32-konnectivity-ca" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442540 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce941b3a-da84-4b86-944c-89f9f464a27f" volumeName="kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442565 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e49a2d50-dc82-4983-9425-053dfe903ddd" volumeName="kubernetes.io/configmap/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-config" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442578 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6924bea-83c3-443f-a95d-581edf9aaf7b" volumeName="kubernetes.io/projected/e6924bea-83c3-443f-a95d-581edf9aaf7b-kube-api-access-mjqft" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442603 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4873e537-31a8-4e6d-86bd-c54024d94c39" volumeName="kubernetes.io/projected/4873e537-31a8-4e6d-86bd-c54024d94c39-kube-api-access-4rd6p" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442624 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6" volumeName="kubernetes.io/secret/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442638 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ed572b5-9b0c-4924-af2f-87703ed1b5e6" volumeName="kubernetes.io/projected/7ed572b5-9b0c-4924-af2f-87703ed1b5e6-kube-api-access-98zhq" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442653 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6f98741-1d48-4124-9acb-8c0bf31dbe60" volumeName="kubernetes.io/configmap/b6f98741-1d48-4124-9acb-8c0bf31dbe60-config-volume" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442674 5188 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29050b65-73ea-446c-98b6-76ce5438f0c9" volumeName="kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6" seLinuxMountContext="" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442687 5188 reconstruct.go:97] "Volume reconstruction finished" Oct 31 21:15:53.443283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.442696 5188 reconciler.go:29] "Reconciler: start to sync state" Oct 31 21:15:53.455634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.455603 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/1.log" Oct 31 21:15:53.455796 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.455699 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/1.log" Oct 31 21:15:53.463470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.462918 5188 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 21:15:53.465037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.464689 5188 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 21:15:53.465037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.464715 5188 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 21:15:53.465037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.464740 5188 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 21:15:53.465037 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:53.464785 5188 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 21:15:53.475145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.474252 5188 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:53.475145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.474478 5188 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 21:15:53.477892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.477869 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/1.log" Oct 31 21:15:53.478008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.477906 5188 generic.go:356] "Generic (PLEG): container finished" podID="1b23775a-33a5-47b0-ac74-fb3862912542" containerID="0728d08f5fd76e870a74999860ecc1ce866ab9547dabb2fedb55a89978d2fab1" exitCode=255 Oct 31 21:15:53.481748 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.481478 5188 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:53.492042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.492020 5188 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:53.493026 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.492998 5188 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:53.496744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.496712 5188 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="b6692948d28a2cf0b45ae0d23a05ba8035a9f89ec279aa95ce55bee2db96cab2" exitCode=0 Oct 31 21:15:53.496744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.496742 5188 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="8a26ab298fcf190d8b2a57c31530f3b6236825d2472274f3e7dc0f98f09a3b85" exitCode=0 Oct 31 21:15:53.496938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.496756 5188 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="c37852e3129d121a35e61aafe9441bedbc60146a0e305682948435acb040601f" exitCode=0 Oct 31 21:15:53.496938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.496766 5188 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="191aa7f4f532eb81a807e9315ca8b35f2744c0f442338b9f859f4244dfd36cc1" exitCode=0 Oct 31 21:15:53.496938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.496772 5188 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="57aaa56b99c5215263015c045823720ad6277d38dd19c662874e51cbdc185d4f" exitCode=0 Oct 31 21:15:53.496938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.496777 5188 generic.go:356] "Generic (PLEG): container finished" podID="a8bb99e2-7626-47ee-b7b6-6b45e954cf4a" containerID="536190f352e88c88cbb3f2c0a2cb1d29dacbea1ced75b9e8ffcf8ba2887014b0" exitCode=0 Oct 31 21:15:53.522366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.522272 5188 generic.go:356] "Generic (PLEG): container finished" podID="76456f0ae0dd752db0186d1d3e70e3aa" containerID="ad29c852c4ea2d970962908954294f93db2665e7f36812d7c047ab5b0de30e58" exitCode=0 Oct 31 21:15:53.542619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.542593 5188 manager.go:324] Recovery completed Oct 31 21:15:53.564985 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:53.564957 5188 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 21:15:53.566986 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.566966 5188 cpu_manager.go:226] "Starting" policy="none" Oct 31 21:15:53.567042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.566986 5188 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 21:15:53.567042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567012 5188 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 21:15:53.567205 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567183 5188 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 21:15:53.567710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567204 5188 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 21:15:53.567743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567715 5188 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 21:15:53.567743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567722 5188 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 21:15:53.567743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567729 5188 policy_none.go:50] "Start" Oct 31 21:15:53.567818 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567751 5188 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 21:15:53.567818 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567768 5188 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 21:15:53.567928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567918 5188 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 21:15:53.567958 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567929 5188 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 21:15:53.567958 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.567936 5188 policy_none.go:44] "Start" Oct 31 21:15:53.572341 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.572302 5188 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 21:15:53.575188 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.575172 5188 manager.go:341] "Starting Device Plugin manager" Oct 31 21:15:53.575264 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:53.575209 5188 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 21:15:53.575264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.575221 5188 server.go:87] "Starting device plugin registration server" Oct 31 21:15:53.575615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.575601 5188 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 21:15:53.575649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.575620 5188 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 21:15:53.575722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.575705 5188 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 21:15:53.575849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.575814 5188 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 21:15:53.575899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.575860 5188 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 21:15:53.575899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.575868 5188 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 21:15:53.575997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.575926 5188 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T21:15:53.575845224Z","UUID":"24ecc97c-791f-4aea-8194-67aa8b10b59d","Handler":null,"Name":"","Endpoint":""} Oct 31 21:15:53.577406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.577263 5188 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 21:15:53.577406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.577304 5188 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 21:15:53.577589 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:53.577567 5188 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 21:15:53.676421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.676383 5188 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:53.677773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.677752 5188 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:53.677866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.677786 5188 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:53.677866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.677804 5188 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-21.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:53.677866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.677851 5188 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.707416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.707385 5188 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.707568 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.707468 5188 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.765959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.765890 5188 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal"] Oct 31 21:15:53.766738 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.766708 5188 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.766837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.766724 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" event={"ID":"76456f0ae0dd752db0186d1d3e70e3aa","Type":"ContainerStarted","Data":"b35f576ff1f943ae20965f9794d295ae98788eaed83a95b9db9e18a5ba92ef87"} Oct 31 21:15:53.766837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.766767 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" event={"ID":"76456f0ae0dd752db0186d1d3e70e3aa","Type":"ContainerDied","Data":"ad29c852c4ea2d970962908954294f93db2665e7f36812d7c047ab5b0de30e58"} Oct 31 21:15:53.766837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.766782 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" event={"ID":"76456f0ae0dd752db0186d1d3e70e3aa","Type":"ContainerStarted","Data":"24ad4aaa02fcb3c760a3fe8813664d0215626be148def102ca59e19d7e3308f5"} Oct 31 21:15:53.766837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.766814 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" event={"ID":"d208acf97ac3e1709a838235c7640507","Type":"ContainerStarted","Data":"91d8b0f6d2263de09dde978a9718c1e7d9ccdab329624d7e8273aaab4e475d9a"} Oct 31 21:15:53.766837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.766828 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" event={"ID":"d208acf97ac3e1709a838235c7640507","Type":"ContainerStarted","Data":"4397200e77b1c2a7f7a1a3354f9066a6a7193afa7d85f8ab9f5c97e84ebedf11"} Oct 31 21:15:53.767015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.766933 5188 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.790564 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.790481 5188 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:15:53.790564 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:53.790546 5188 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.791393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.791382 5188 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:15:53.791484 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:53.791408 5188 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.876800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.876761 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.876800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.876799 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.877002 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.876825 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d208acf97ac3e1709a838235c7640507-config\") pod \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" (UID: \"d208acf97ac3e1709a838235c7640507\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.977454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.977410 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.977614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.977492 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.977614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.977504 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.977614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.977537 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d208acf97ac3e1709a838235c7640507-config\") pod \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" (UID: \"d208acf97ac3e1709a838235c7640507\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.977614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.977555 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76456f0ae0dd752db0186d1d3e70e3aa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" (UID: \"76456f0ae0dd752db0186d1d3e70e3aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:53.977614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:53.977576 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d208acf97ac3e1709a838235c7640507-config\") pod \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" (UID: \"d208acf97ac3e1709a838235c7640507\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:54.413425 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.413388 5188 apiserver.go:52] "Watching apiserver" Oct 31 21:15:54.469305 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.469260 5188 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 21:15:54.470527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.470476 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c58487768-swgvs","openshift-console-operator/console-operator-7cd56b8dbc-cg8tz","openshift-cluster-node-tuning-operator/tuned-mrskg","openshift-multus/multus-jxbcm","openshift-ovn-kubernetes/ovnkube-node-8nl7t","openshift-image-registry/image-registry-574b9c7469-m4qfg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm","openshift-network-operator/iptables-alerter-x47n8","kube-system/kas-connection-checker-685895c7c9-w96gb","openshift-insights/insights-runtime-extractor-dhxqp","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql","openshift-insights/insights-operator-7c4986d449-bnbr9","kube-system/global-pull-secret-syncer-whzgx","openshift-ingress/router-default-68d4c65fcc-47wdn","openshift-multus/network-metrics-daemon-zc54c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg","openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x","openshift-ingress-canary/ingress-canary-mb4ql","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6","openshift-multus/multus-additional-cni-plugins-2l8b7","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr","openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk","kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal","openshift-service-ca/service-ca-5fffc97bdf-8qdzg","kube-system/konnectivity-agent-9kbsn","openshift-image-registry/node-ca-42vxn","openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j","kube-system/kas-connection-checker-685895c7c9-cps44","openshift-dns/node-resolver-vx79w","kube-system/kas-connection-checker-685895c7c9-7fh65","openshift-network-console/networking-console-plugin-86c478bb88-sh7vx","openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd","openshift-dns/dns-default-wj49j","openshift-network-diagnostics/network-check-target-tpl7t","openshift-insights/periodic-gathering-vk6tn-jprcv"] Oct 31 21:15:54.472408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.472380 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.473629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.473606 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:54.474250 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.474231 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:54.474932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.474913 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.475305 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.475288 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:54.479146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.479128 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:54.484267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.484227 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.484416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.484405 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.484534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.484518 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.484584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.484536 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 21:15:54.484629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.484605 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.484674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.484642 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.484861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.484845 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 21:15:54.484914 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.484870 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.484959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.484932 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.485132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.485115 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.485318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.485299 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.485854 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.485832 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 21:15:54.486075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.486059 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7pbqt\"" Oct 31 21:15:54.486143 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.486120 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 21:15:54.486225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.486122 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 21:15:54.486364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.486347 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nt72v\"" Oct 31 21:15:54.486418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.486379 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 21:15:54.486694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.486675 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qc9xz\"" Oct 31 21:15:54.486905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.486876 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gpdvx\"" Oct 31 21:15:54.487122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.487103 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.488749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.488724 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.494289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.494257 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 21:15:54.494418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.494340 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 21:15:54.494418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.494368 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5xwjc\"" Oct 31 21:15:54.494418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.494404 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cs6xw\"" Oct 31 21:15:54.494857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.494842 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 21:15:54.494933 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.494914 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.494987 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.494976 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.495044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495016 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 21:15:54.495084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495048 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xsvd6\"" Oct 31 21:15:54.495503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495474 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.495603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495527 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 21:15:54.495603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495545 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 21:15:54.495603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495578 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.495603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495599 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 21:15:54.495831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495688 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.495831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495689 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 21:15:54.495831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495752 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 21:15:54.495831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495752 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.495831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495769 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdqcm\"" Oct 31 21:15:54.495831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495773 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.495831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495755 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 21:15:54.495831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495768 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 21:15:54.495831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495753 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 21:15:54.496109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495857 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 21:15:54.496109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495877 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.496109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495940 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-48hbl\"" Oct 31 21:15:54.496109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.495984 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 21:15:54.496109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.496079 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.496301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.496184 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-nhmjw\"" Oct 31 21:15:54.496521 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.496504 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 21:15:54.497013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.497000 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 21:15:54.500140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.500120 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.500751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.500732 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.500937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.500776 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.504171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504133 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 21:15:54.504365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504270 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 21:15:54.504365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504344 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 21:15:54.504529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504385 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 21:15:54.504529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504396 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 21:15:54.504529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504400 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nvc4l\"" Oct 31 21:15:54.504529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504468 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m2xrj\"" Oct 31 21:15:54.504529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504473 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 21:15:54.504807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504550 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.504807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504565 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.504807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504587 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 21:15:54.504807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504625 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 21:15:54.504974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.504925 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g9h45\"" Oct 31 21:15:54.505044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505029 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.505226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505199 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.505226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505216 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 21:15:54.505373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505218 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.505373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505239 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.505373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505257 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 21:15:54.505373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505294 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.505373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505357 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.505373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505363 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-rdjnn\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505358 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505397 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-866d2\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505406 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505358 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-28gfg\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505453 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p85m4\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505459 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505475 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505483 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvv8l\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505485 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505485 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505509 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ddmwn\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505518 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505533 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505538 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505539 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505548 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505579 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6mmhq\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505582 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9gmld\"" Oct 31 21:15:54.505639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505648 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505651 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505732 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vddbx\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505782 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j68t2\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505924 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.505978 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506023 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506095 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506110 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dwlts\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506123 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506132 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506161 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflcb\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506207 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506376 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.506415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506400 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 21:15:54.506842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506445 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nq8jq\"" Oct 31 21:15:54.506842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506479 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 21:15:54.506842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506528 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 21:15:54.506842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506578 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ngrs9\"" Oct 31 21:15:54.506842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506673 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 21:15:54.506842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506762 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:54.506842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.506817 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qtvpr\"" Oct 31 21:15:54.529634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.529606 5188 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:54.529782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.529707 5188 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:54.532103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.532085 5188 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 21:15:54.550771 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.550737 5188 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:15:54.550926 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.550805 5188 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-21.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-21.ec2.internal" Oct 31 21:15:54.553458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.553436 5188 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:15:54.553563 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.553494 5188 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-21.ec2.internal" Oct 31 21:15:54.579640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579602 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.579640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579642 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:54.579902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579661 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:54.579902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579682 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-log-socket\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.579902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579730 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-netns\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.579902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579753 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-multus-daemon-config\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.579902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579779 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:54.579902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579803 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-snapshots\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.579902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579809 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:54.579902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579853 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-kubernetes\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.580206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579928 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-snapshots\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.580206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.579995 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.580206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580043 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-config\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:54.580206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580050 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:54.580206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580162 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-multus-daemon-config\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580244 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-sys-fs\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580284 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6805be97-0a2e-490c-a50c-6700f15c4abc-tmp-dir\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580311 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-config\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580317 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-tmp\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580387 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5f5z\" (UniqueName: \"kubernetes.io/projected/25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5-kube-api-access-p5f5z\") pod \"migrator-66df5d496b-qrc2x\" (UID: \"25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580395 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6805be97-0a2e-490c-a50c-6700f15c4abc-tmp-dir\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580416 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-k8s-cni-cncf-io\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580425 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580447 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qmg\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.580453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580452 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-tmp\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580484 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-system-cni-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580533 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw9ch\" (UniqueName: \"kubernetes.io/projected/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-kube-api-access-cw9ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580564 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-var-lib-kubelet\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580607 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580630 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b23775a-33a5-47b0-ac74-fb3862912542-serving-cert\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580676 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-default-certificate\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580712 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac6429fb-48a7-4691-9079-4ed168f1da32-agent-certs\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580740 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b23775a-33a5-47b0-ac74-fb3862912542-serving-cert\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580773 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580815 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-slash\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580836 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpsd4\" (UniqueName: \"kubernetes.io/projected/6805be97-0a2e-490c-a50c-6700f15c4abc-kube-api-access-vpsd4\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580853 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580872 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f98741-1d48-4124-9acb-8c0bf31dbe60-config-volume\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580900 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580920 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580926 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/595edb9d-3370-41bd-82c3-af8df664ad90-signing-key\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:54.581015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.580993 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-default-certificate\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581028 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ac6429fb-48a7-4691-9079-4ed168f1da32-agent-certs\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581044 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j274g\" (UniqueName: \"kubernetes.io/projected/595edb9d-3370-41bd-82c3-af8df664ad90-kube-api-access-j274g\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581083 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581105 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b6f98741-1d48-4124-9acb-8c0bf31dbe60-config-volume\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581113 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b445d\" (UniqueName: \"kubernetes.io/projected/d2a47228-9b40-4833-bb84-970f4782e67b-kube-api-access-b445d\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581152 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcxl\" (UniqueName: \"kubernetes.io/projected/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-kube-api-access-hpcxl\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581183 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581206 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-run\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581231 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581232 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5kk\" (UniqueName: \"kubernetes.io/projected/9f8d2547-a263-47ff-aaa9-780ea67176a5-kube-api-access-gs5kk\") pod \"kas-connection-checker-685895c7c9-w96gb\" (UID: \"9f8d2547-a263-47ff-aaa9-780ea67176a5\") " pod="kube-system/kas-connection-checker-685895c7c9-w96gb" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581232 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581253 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/595edb9d-3370-41bd-82c3-af8df664ad90-signing-key\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581299 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581369 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-cnibin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581430 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581437 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581496 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cnibin\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581519 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581553 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-modprobe-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581582 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfgnv\" (UniqueName: \"kubernetes.io/projected/a4772ac2-3076-4099-89da-cb3be3d4387c-kube-api-access-tfgnv\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581611 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjqft\" (UniqueName: \"kubernetes.io/projected/e6924bea-83c3-443f-a95d-581edf9aaf7b-kube-api-access-mjqft\") pod \"network-check-source-6f78d59f99-cbc4j\" (UID: \"e6924bea-83c3-443f-a95d-581edf9aaf7b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581647 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rxlh\" (UniqueName: \"kubernetes.io/projected/e49a2d50-dc82-4983-9425-053dfe903ddd-kube-api-access-4rxlh\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581669 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581690 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581716 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7245fb80-34d5-471b-984e-7fbfe2b1113c-data-volume\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581768 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac6429fb-48a7-4691-9079-4ed168f1da32-konnectivity-ca\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581824 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-dbus\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581827 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7245fb80-34d5-471b-984e-7fbfe2b1113c-data-volume\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581875 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-script-lib\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.581921 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhwr\" (UniqueName: \"kubernetes.io/projected/47354b4e-2a35-40be-a539-07dad787bd97-kube-api-access-8jhwr\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582005 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ac6429fb-48a7-4691-9079-4ed168f1da32-konnectivity-ca\") pod \"konnectivity-agent-9kbsn\" (UID: \"ac6429fb-48a7-4691-9079-4ed168f1da32\") " pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582027 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6805be97-0a2e-490c-a50c-6700f15c4abc-hosts-file\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582053 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxpcl\" (UniqueName: \"kubernetes.io/projected/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-kube-api-access-bxpcl\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582071 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582093 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whfjx\" (UniqueName: \"kubernetes.io/projected/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-api-access-whfjx\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582118 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcwz\" (UniqueName: \"kubernetes.io/projected/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-kube-api-access-bjcwz\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582136 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-script-lib\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582142 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/add13542-315e-4dde-a5fa-728aaf6fdc6c-cert\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582180 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-bin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582211 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-stats-auth\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582219 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582234 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-lib-modules\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582355 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582382 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98zhq\" (UniqueName: \"kubernetes.io/projected/7ed572b5-9b0c-4924-af2f-87703ed1b5e6-kube-api-access-98zhq\") pod \"kas-connection-checker-685895c7c9-cps44\" (UID: \"7ed572b5-9b0c-4924-af2f-87703ed1b5e6\") " pod="kube-system/kas-connection-checker-685895c7c9-cps44" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582392 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/add13542-315e-4dde-a5fa-728aaf6fdc6c-cert\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582408 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725ms\" (UniqueName: \"kubernetes.io/projected/626d59af-b777-449a-87cc-23e763dd495f-kube-api-access-725ms\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582434 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-kubelet\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582479 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-stats-auth\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582533 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hljm\" (UniqueName: \"kubernetes.io/projected/1b23775a-33a5-47b0-ac74-fb3862912542-kube-api-access-7hljm\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582562 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-host\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582661 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fwg\" (UniqueName: \"kubernetes.io/projected/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-kube-api-access-t7fwg\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582689 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582728 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnm6\" (UniqueName: \"kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582805 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcnhc\" (UniqueName: \"kubernetes.io/projected/b6f98741-1d48-4124-9acb-8c0bf31dbe60-kube-api-access-pcnhc\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582831 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582834 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582855 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-iptables-alerter-script\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582944 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582966 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582982 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/672a2ce2-d45d-4f0c-a75a-b018105640aa-serviceca\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.582997 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583008 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583015 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-host-slash\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583039 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7245fb80-34d5-471b-984e-7fbfe2b1113c-crio-socket\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583055 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583106 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-ovn\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583118 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-iptables-alerter-script\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583130 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583180 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/672a2ce2-d45d-4f0c-a75a-b018105640aa-serviceca\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583185 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583220 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31682289-5cba-480d-88a4-c71e9cd6244b-serving-cert\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583240 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583260 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-config\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583284 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-conf-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583287 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583293 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2a47228-9b40-4833-bb84-970f4782e67b-service-ca-bundle\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583310 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-sys\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583396 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583407 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-kubelet\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583435 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-env-overrides\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583467 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-system-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583502 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovnkube-config\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583520 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/31682289-5cba-480d-88a4-c71e9cd6244b-serving-cert\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583555 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583572 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583589 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583600 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-env-overrides\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583607 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583665 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583698 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-kubelet-config\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583727 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-bin\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583784 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583841 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583878 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-hostroot\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583905 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-var-lib-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583928 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583947 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/672a2ce2-d45d-4f0c-a75a-b018105640aa-host\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.583994 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwtqf\" (UniqueName: \"kubernetes.io/projected/add13542-315e-4dde-a5fa-728aaf6fdc6c-kube-api-access-mwtqf\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584027 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584055 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584088 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584096 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584115 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31682289-5cba-480d-88a4-c71e9cd6244b-config\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584164 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/595edb9d-3370-41bd-82c3-af8df664ad90-signing-cabundle\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584203 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktw9c\" (UniqueName: \"kubernetes.io/projected/4bf7148c-69be-45a9-bd56-7457a3929df8-kube-api-access-ktw9c\") pod \"kas-connection-checker-685895c7c9-7fh65\" (UID: \"4bf7148c-69be-45a9-bd56-7457a3929df8\") " pod="kube-system/kas-connection-checker-685895c7c9-7fh65" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584224 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-tmp\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584240 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-cni-binary-copy\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584258 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58tf\" (UniqueName: \"kubernetes.io/projected/3616b3de-b6e6-497f-8546-8fef766c309f-kube-api-access-f58tf\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584282 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927v6\" (UniqueName: \"kubernetes.io/projected/f7d03911-7706-47b8-928e-6ffea21fdb26-kube-api-access-927v6\") pod \"volume-data-source-validator-58c9f5f46d-swfql\" (UID: \"f7d03911-7706-47b8-928e-6ffea21fdb26\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584313 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584356 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2a47228-9b40-4833-bb84-970f4782e67b-tmp\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584421 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/31682289-5cba-480d-88a4-c71e9cd6244b-config\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584436 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/595edb9d-3370-41bd-82c3-af8df664ad90-signing-cabundle\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584456 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-systemd\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584526 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584564 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47354b4e-2a35-40be-a539-07dad787bd97-cni-binary-copy\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584575 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584565 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-trusted-ca\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584620 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584733 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-socket-dir-parent\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584822 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-multus-certs\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584857 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584919 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b23775a-33a5-47b0-ac74-fb3862912542-trusted-ca\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584927 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t57pd\" (UniqueName: \"kubernetes.io/projected/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-kube-api-access-t57pd\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584979 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-tuned\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.584994 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72751047-bc3c-4f2d-9655-e23d5211d346-original-pull-secret\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585042 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-node-log\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585069 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-tuned\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585100 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-etc-kubernetes\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585108 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585124 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/31682289-5cba-480d-88a4-c71e9cd6244b-kube-api-access-lxlcr\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585154 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a47228-9b40-4833-bb84-970f4782e67b-serving-cert\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.597929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585207 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-netns\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585228 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-device-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585246 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f98741-1d48-4124-9acb-8c0bf31dbe60-tmp-dir\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585267 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585294 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585294 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2a47228-9b40-4833-bb84-970f4782e67b-serving-cert\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585317 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6f98741-1d48-4124-9acb-8c0bf31dbe60-tmp-dir\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585336 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-etc-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585389 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585414 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-netd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585431 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-multus\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585454 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585480 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585499 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455ft\" (UniqueName: \"kubernetes.io/projected/672a2ce2-d45d-4f0c-a75a-b018105640aa-kube-api-access-455ft\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585573 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585627 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585667 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585697 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysconfig\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585719 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-conf\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585824 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585849 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-systemd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585826 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cni-binary-copy\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585873 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585907 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rd6p\" (UniqueName: \"kubernetes.io/projected/4873e537-31a8-4e6d-86bd-c54024d94c39-kube-api-access-4rd6p\") pod \"migrator-66df5d496b-g7thd\" (UID: \"4873e537-31a8-4e6d-86bd-c54024d94c39\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585926 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-registration-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585952 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-systemd-units\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585971 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vtd\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585985 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.585994 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-socket-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.602033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.586238 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-ovn-node-metrics-cert\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.605286 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.605247 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw9ch\" (UniqueName: \"kubernetes.io/projected/21a28634-04b5-4d1b-bf14-6bdc3b894f2d-kube-api-access-cw9ch\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6\" (UID: \"21a28634-04b5-4d1b-bf14-6bdc3b894f2d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" Oct 31 21:15:54.605506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.605485 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpsd4\" (UniqueName: \"kubernetes.io/projected/6805be97-0a2e-490c-a50c-6700f15c4abc-kube-api-access-vpsd4\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:54.605554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.605530 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5f5z\" (UniqueName: \"kubernetes.io/projected/25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5-kube-api-access-p5f5z\") pod \"migrator-66df5d496b-qrc2x\" (UID: \"25bcf1fd-b559-4c7c-b6af-22ff9d86e6a5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qrc2x" Oct 31 21:15:54.605595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.605585 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j274g\" (UniqueName: \"kubernetes.io/projected/595edb9d-3370-41bd-82c3-af8df664ad90-kube-api-access-j274g\") pod \"service-ca-5fffc97bdf-8qdzg\" (UID: \"595edb9d-3370-41bd-82c3-af8df664ad90\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8qdzg" Oct 31 21:15:54.605649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.605627 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qmg\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.605916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.605902 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc8sb\" (UniqueName: \"kubernetes.io/projected/c929e38e-147c-41f9-ad34-b0af48044e09-kube-api-access-fc8sb\") pod \"network-check-target-tpl7t\" (UID: \"c929e38e-147c-41f9-ad34-b0af48044e09\") " pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:54.606493 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.606474 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5kk\" (UniqueName: \"kubernetes.io/projected/9f8d2547-a263-47ff-aaa9-780ea67176a5-kube-api-access-gs5kk\") pod \"kas-connection-checker-685895c7c9-w96gb\" (UID: \"9f8d2547-a263-47ff-aaa9-780ea67176a5\") " pod="kube-system/kas-connection-checker-685895c7c9-w96gb" Oct 31 21:15:54.606769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.606753 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b445d\" (UniqueName: \"kubernetes.io/projected/d2a47228-9b40-4833-bb84-970f4782e67b-kube-api-access-b445d\") pod \"insights-operator-7c4986d449-bnbr9\" (UID: \"d2a47228-9b40-4833-bb84-970f4782e67b\") " pod="openshift-insights/insights-operator-7c4986d449-bnbr9" Oct 31 21:15:54.607036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.607020 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjqft\" (UniqueName: \"kubernetes.io/projected/e6924bea-83c3-443f-a95d-581edf9aaf7b-kube-api-access-mjqft\") pod \"network-check-source-6f78d59f99-cbc4j\" (UID: \"e6924bea-83c3-443f-a95d-581edf9aaf7b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cbc4j" Oct 31 21:15:54.608531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.608506 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-725ms\" (UniqueName: \"kubernetes.io/projected/626d59af-b777-449a-87cc-23e763dd495f-kube-api-access-725ms\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.608869 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.608816 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whfjx\" (UniqueName: \"kubernetes.io/projected/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-api-access-whfjx\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.608869 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.608815 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rxlh\" (UniqueName: \"kubernetes.io/projected/e49a2d50-dc82-4983-9425-053dfe903ddd-kube-api-access-4rxlh\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:54.609085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.608905 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.609085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.609045 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcxl\" (UniqueName: \"kubernetes.io/projected/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-kube-api-access-hpcxl\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:54.609380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.609359 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fwg\" (UniqueName: \"kubernetes.io/projected/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-kube-api-access-t7fwg\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:54.609497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.609481 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hljm\" (UniqueName: \"kubernetes.io/projected/1b23775a-33a5-47b0-ac74-fb3862912542-kube-api-access-7hljm\") pod \"console-operator-7cd56b8dbc-cg8tz\" (UID: \"1b23775a-33a5-47b0-ac74-fb3862912542\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:54.610378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.610354 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.610477 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.610402 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rd6p\" (UniqueName: \"kubernetes.io/projected/4873e537-31a8-4e6d-86bd-c54024d94c39-kube-api-access-4rd6p\") pod \"migrator-66df5d496b-g7thd\" (UID: \"4873e537-31a8-4e6d-86bd-c54024d94c39\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g7thd" Oct 31 21:15:54.610514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.610494 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t57pd\" (UniqueName: \"kubernetes.io/projected/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-kube-api-access-t57pd\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.610892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.610870 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlcr\" (UniqueName: \"kubernetes.io/projected/31682289-5cba-480d-88a4-c71e9cd6244b-kube-api-access-lxlcr\") pod \"service-ca-operator-544d749c8c-p7ctk\" (UID: \"31682289-5cba-480d-88a4-c71e9cd6244b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" Oct 31 21:15:54.610939 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.610905 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnm6\" (UniqueName: \"kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6\") pod \"periodic-gathering-vk6tn-jprcv\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:15:54.611035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.611015 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98zhq\" (UniqueName: \"kubernetes.io/projected/7ed572b5-9b0c-4924-af2f-87703ed1b5e6-kube-api-access-98zhq\") pod \"kas-connection-checker-685895c7c9-cps44\" (UID: \"7ed572b5-9b0c-4924-af2f-87703ed1b5e6\") " pod="kube-system/kas-connection-checker-685895c7c9-cps44" Oct 31 21:15:54.611589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.611572 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhwr\" (UniqueName: \"kubernetes.io/projected/47354b4e-2a35-40be-a539-07dad787bd97-kube-api-access-8jhwr\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.611682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.611667 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktw9c\" (UniqueName: \"kubernetes.io/projected/4bf7148c-69be-45a9-bd56-7457a3929df8-kube-api-access-ktw9c\") pod \"kas-connection-checker-685895c7c9-7fh65\" (UID: \"4bf7148c-69be-45a9-bd56-7457a3929df8\") " pod="kube-system/kas-connection-checker-685895c7c9-7fh65" Oct 31 21:15:54.613059 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.613036 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcwz\" (UniqueName: \"kubernetes.io/projected/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-kube-api-access-bjcwz\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.613853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.613830 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58tf\" (UniqueName: \"kubernetes.io/projected/3616b3de-b6e6-497f-8546-8fef766c309f-kube-api-access-f58tf\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:54.614023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.614005 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-455ft\" (UniqueName: \"kubernetes.io/projected/672a2ce2-d45d-4f0c-a75a-b018105640aa-kube-api-access-455ft\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:54.614203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.614182 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfgnv\" (UniqueName: \"kubernetes.io/projected/a4772ac2-3076-4099-89da-cb3be3d4387c-kube-api-access-tfgnv\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.614440 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.614426 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxpcl\" (UniqueName: \"kubernetes.io/projected/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-kube-api-access-bxpcl\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.614764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.614743 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcnhc\" (UniqueName: \"kubernetes.io/projected/b6f98741-1d48-4124-9acb-8c0bf31dbe60-kube-api-access-pcnhc\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.614855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.614836 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vtd\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.615375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.615356 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwtqf\" (UniqueName: \"kubernetes.io/projected/add13542-315e-4dde-a5fa-728aaf6fdc6c-kube-api-access-mwtqf\") pod \"ingress-canary-mb4ql\" (UID: \"add13542-315e-4dde-a5fa-728aaf6fdc6c\") " pod="openshift-ingress-canary/ingress-canary-mb4ql" Oct 31 21:15:54.615720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.615702 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-927v6\" (UniqueName: \"kubernetes.io/projected/f7d03911-7706-47b8-928e-6ffea21fdb26-kube-api-access-927v6\") pod \"volume-data-source-validator-58c9f5f46d-swfql\" (UID: \"f7d03911-7706-47b8-928e-6ffea21fdb26\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-swfql" Oct 31 21:15:54.686954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.686862 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-host\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.686954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.686901 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.686954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.686919 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-host-slash\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:54.686954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.686941 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7245fb80-34d5-471b-984e-7fbfe2b1113c-crio-socket\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.686965 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.686988 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-ovn\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.686992 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-host\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687001 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43bf6948-d5fa-4e37-8f85-d2d9dd9d4936-host-slash\") pod \"iptables-alerter-x47n8\" (UID: \"43bf6948-d5fa-4e37-8f85-d2d9dd9d4936\") " pod="openshift-network-operator/iptables-alerter-x47n8" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687013 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687020 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7245fb80-34d5-471b-984e-7fbfe2b1113c-crio-socket\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687053 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-conf-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687064 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-ovn\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687063 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687070 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687096 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-sys\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687107 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-conf-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687141 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-kubelet\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687171 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-sys\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687199 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-kubelet\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687206 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-system-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687240 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687246 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-system-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687266 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-kubelet-config\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687282 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-bin\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.687281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687297 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-hostroot\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687317 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-var-lib-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687352 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/672a2ce2-d45d-4f0c-a75a-b018105640aa-host\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687366 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-bin\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687379 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687396 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-hostroot\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687403 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687399 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-var-lib-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687411 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/672a2ce2-d45d-4f0c-a75a-b018105640aa-host\") pod \"node-ca-42vxn\" (UID: \"672a2ce2-d45d-4f0c-a75a-b018105640aa\") " pod="openshift-image-registry/node-ca-42vxn" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687365 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-kubelet-config\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687444 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-systemd\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687474 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.687492 5188 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687512 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-socket-dir-parent\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687537 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-multus-certs\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687539 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-cni-dir\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687538 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7245fb80-34d5-471b-984e-7fbfe2b1113c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687550 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.687578 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:55.187556082 +0000 UTC m=+1.880513475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : secret "insights-runtime-extractor-tls" not found Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687579 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-systemd\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687585 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-multus-socket-dir-parent\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687585 5188 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687585 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-multus-certs\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687614 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-node-log\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687638 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-etc-kubernetes\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687657 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-netns\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687669 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-node-log\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687693 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-device-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687718 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687712 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-etc-kubernetes\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687745 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687772 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-etc-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687796 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687802 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687788 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-netns\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687819 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-netd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687812 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-device-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687843 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-multus\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687848 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-etc-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687869 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-cni-netd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687872 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687871 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687916 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-multus\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687947 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.687988 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysconfig\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688011 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-conf\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688036 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-systemd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.688047 5188 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688061 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-registration-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.688116 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:55.1880966 +0000 UTC m=+1.881053984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : secret "metrics-daemon-secret" not found Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688123 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-systemd-units\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688159 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-socket-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688187 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688214 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-log-socket\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688237 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-netns\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688256 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysctl-conf\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.688228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688264 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-kubernetes\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688293 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-sys-fs\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688307 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-sysconfig\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688343 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-k8s-cni-cncf-io\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688373 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-system-cni-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688400 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-var-lib-kubelet\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688423 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688453 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-slash\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688488 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688511 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-run\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688538 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-cnibin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688561 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688586 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cnibin\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688611 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-modprobe-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688643 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688713 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-etc-selinux\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688760 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-host-slash\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688807 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-cnibin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.688886 5188 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688902 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-sys-fs\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.688944 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert podName:0bd131db-e6d0-46c6-9a4b-85d3b541a9ec nodeName:}" failed. No retries permitted until 2026-10-31 21:15:55.188928422 +0000 UTC m=+1.881885797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sh7vx" (UID: "0bd131db-e6d0-46c6-9a4b-85d3b541a9ec") : secret "networking-console-plugin-cert" not found Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688905 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-cnibin\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688942 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-log-socket\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688963 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-run\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688995 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-netns\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.688998 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8bb99e2-7626-47ee-b7b6-6b45e954cf4a-system-cni-dir\") pod \"multus-additional-cni-plugins-2l8b7\" (UID: \"a8bb99e2-7626-47ee-b7b6-6b45e954cf4a\") " pod="openshift-multus/multus-additional-cni-plugins-2l8b7" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689014 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-kubernetes\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689047 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-systemd\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689056 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-systemd-units\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689083 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-run-k8s-cni-cncf-io\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689118 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-var-lib-kubelet\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689128 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-registration-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689155 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-etc-modprobe-d\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689167 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/626d59af-b777-449a-87cc-23e763dd495f-socket-dir\") pod \"aws-ebs-csi-driver-node-bsjcg\" (UID: \"626d59af-b777-449a-87cc-23e763dd495f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bsjcg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.689186 5188 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689231 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689263 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-dbus\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.689305 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6805be97-0a2e-490c-a50c-6700f15c4abc-hosts-file\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:54.689841 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls podName:c01f3957-4a49-41ba-812b-ea2e74d3a7a8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:55.189821799 +0000 UTC m=+1.882779185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8r9vm" (UID: "c01f3957-4a49-41ba-812b-ea2e74d3a7a8") : secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690014 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-service-ca-bundle\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690147 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-bin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690179 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-lib-modules\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690182 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6805be97-0a2e-490c-a50c-6700f15c4abc-hosts-file\") pod \"node-resolver-vx79w\" (UID: \"6805be97-0a2e-490c-a50c-6700f15c4abc\") " pod="openshift-dns/node-resolver-vx79w" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690202 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690231 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-kubelet\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690307 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-kubelet\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690369 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47354b4e-2a35-40be-a539-07dad787bd97-host-var-lib-cni-bin\") pod \"multus-jxbcm\" (UID: \"47354b4e-2a35-40be-a539-07dad787bd97\") " pod="openshift-multus/multus-jxbcm" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690458 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4772ac2-3076-4099-89da-cb3be3d4387c-lib-modules\") pod \"tuned-mrskg\" (UID: \"a4772ac2-3076-4099-89da-cb3be3d4387c\") " pod="openshift-cluster-node-tuning-operator/tuned-mrskg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690494 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6-run-openvswitch\") pod \"ovnkube-node-8nl7t\" (UID: \"4eb10a24-4ad9-48b0-8f68-3f2fd59d07d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690709 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"image-registry-574b9c7469-m4qfg\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.690777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.690804 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6f98741-1d48-4124-9acb-8c0bf31dbe60-metrics-tls\") pod \"dns-default-wj49j\" (UID: \"b6f98741-1d48-4124-9acb-8c0bf31dbe60\") " pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:54.692900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.691154 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e49a2d50-dc82-4983-9425-053dfe903ddd-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-k4qxr\" (UID: \"e49a2d50-dc82-4983-9425-053dfe903ddd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:54.692900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.691286 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"image-registry-c58487768-swgvs\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.692900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.691517 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72751047-bc3c-4f2d-9655-e23d5211d346-dbus\") pod \"global-pull-secret-syncer-whzgx\" (UID: \"72751047-bc3c-4f2d-9655-e23d5211d346\") " pod="kube-system/global-pull-secret-syncer-whzgx" Oct 31 21:15:54.692900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.692770 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/934a6c09-d6c4-4ae2-afbf-3bc19cc948c8-metrics-certs\") pod \"router-default-68d4c65fcc-47wdn\" (UID: \"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8\") " pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.772841 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.772673 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:54.774306 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.774280 5188 scope.go:122] "RemoveContainer" containerID="0728d08f5fd76e870a74999860ecc1ce866ab9547dabb2fedb55a89978d2fab1" Oct 31 21:15:54.774647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.774628 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" Oct 31 21:15:54.775295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.775276 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:54.785484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.785465 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:54.787554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:54.787536 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:55.050748 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.050715 5188 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:55.115434 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.115154 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr"] Oct 31 21:15:55.120713 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.120689 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-68d4c65fcc-47wdn"] Oct 31 21:15:55.122815 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:55.122784 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934a6c09_d6c4_4ae2_afbf_3bc19cc948c8.slice/crio-e2b6ddfea7302ca41c3b0a9ef43fd169ee87e5e6add42dde05a8ff2dc52f06dc WatchSource:0}: Error finding container e2b6ddfea7302ca41c3b0a9ef43fd169ee87e5e6add42dde05a8ff2dc52f06dc: Status 404 returned error can't find the container with id e2b6ddfea7302ca41c3b0a9ef43fd169ee87e5e6add42dde05a8ff2dc52f06dc Oct 31 21:15:55.157658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.157624 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wj49j"] Oct 31 21:15:55.162033 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:55.162003 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f98741_1d48_4124_9acb_8c0bf31dbe60.slice/crio-131972a69463d6c38cbe885b4268aa88de511d1ae42dd503b7720717a7c0ee6e WatchSource:0}: Error finding container 131972a69463d6c38cbe885b4268aa88de511d1ae42dd503b7720717a7c0ee6e: Status 404 returned error can't find the container with id 131972a69463d6c38cbe885b4268aa88de511d1ae42dd503b7720717a7c0ee6e Oct 31 21:15:55.162162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.162063 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-574b9c7469-m4qfg"] Oct 31 21:15:55.164685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.164663 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:15:55.166263 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:55.166234 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce941b3a_da84_4b86_944c_89f9f464a27f.slice/crio-6564c8f268b71431e88970540ea281ab75021339847e5664015204fbb4616722 WatchSource:0}: Error finding container 6564c8f268b71431e88970540ea281ab75021339847e5664015204fbb4616722: Status 404 returned error can't find the container with id 6564c8f268b71431e88970540ea281ab75021339847e5664015204fbb4616722 Oct 31 21:15:55.186266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.186240 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c58487768-swgvs"] Oct 31 21:15:55.191312 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:55.191290 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e27e3a3_5abc_4fe2_87f6_01dfccdb5d1b.slice/crio-1b0c3a5f660b11252aa59c03ebc12b2d7d3d8e93020ac73fbfe298db76dfc2b3 WatchSource:0}: Error finding container 1b0c3a5f660b11252aa59c03ebc12b2d7d3d8e93020ac73fbfe298db76dfc2b3: Status 404 returned error can't find the container with id 1b0c3a5f660b11252aa59c03ebc12b2d7d3d8e93020ac73fbfe298db76dfc2b3 Oct 31 21:15:55.194774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.194753 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:55.194856 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.194797 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:55.194923 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.194862 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:55.194923 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.194911 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:55.195025 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:55.194920 5188 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:55.195025 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:55.194982 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls podName:c01f3957-4a49-41ba-812b-ea2e74d3a7a8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:56.194961753 +0000 UTC m=+2.887919140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8r9vm" (UID: "c01f3957-4a49-41ba-812b-ea2e74d3a7a8") : secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:55.195025 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:55.194995 5188 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 21:15:55.195174 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:55.195028 5188 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 21:15:55.195174 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:55.195052 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert podName:0bd131db-e6d0-46c6-9a4b-85d3b541a9ec nodeName:}" failed. No retries permitted until 2026-10-31 21:15:56.195035585 +0000 UTC m=+2.887992973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sh7vx" (UID: "0bd131db-e6d0-46c6-9a4b-85d3b541a9ec") : secret "networking-console-plugin-cert" not found Oct 31 21:15:55.195174 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:55.195032 5188 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 21:15:55.195174 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:55.195073 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:56.195059376 +0000 UTC m=+2.888016754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : secret "metrics-daemon-secret" not found Oct 31 21:15:55.195174 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:55.195098 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:56.195088542 +0000 UTC m=+2.888045916 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : secret "insights-runtime-extractor-tls" not found Oct 31 21:15:55.534804 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.534702 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" event={"ID":"e49a2d50-dc82-4983-9425-053dfe903ddd","Type":"ContainerStarted","Data":"4e27cb4cc0bbc7f7be1e394668f8cfc75c97d4a5da9c0d4aa847ce1c58fce07e"} Oct 31 21:15:55.536664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.536627 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" event={"ID":"ce941b3a-da84-4b86-944c-89f9f464a27f","Type":"ContainerStarted","Data":"d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2"} Oct 31 21:15:55.536813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.536687 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" event={"ID":"ce941b3a-da84-4b86-944c-89f9f464a27f","Type":"ContainerStarted","Data":"6564c8f268b71431e88970540ea281ab75021339847e5664015204fbb4616722"} Oct 31 21:15:55.538012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.537979 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wj49j" event={"ID":"b6f98741-1d48-4124-9acb-8c0bf31dbe60","Type":"ContainerStarted","Data":"131972a69463d6c38cbe885b4268aa88de511d1ae42dd503b7720717a7c0ee6e"} Oct 31 21:15:55.539704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.539677 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68d4c65fcc-47wdn" event={"ID":"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8","Type":"ContainerStarted","Data":"37fa865c05f097063d7fffe29be75bd586e6716aa1f22db65c2b1b134b22dca8"} Oct 31 21:15:55.539846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.539719 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-68d4c65fcc-47wdn" event={"ID":"934a6c09-d6c4-4ae2-afbf-3bc19cc948c8","Type":"ContainerStarted","Data":"e2b6ddfea7302ca41c3b0a9ef43fd169ee87e5e6add42dde05a8ff2dc52f06dc"} Oct 31 21:15:55.541478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.541453 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/2.log" Oct 31 21:15:55.541893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.541876 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/1.log" Oct 31 21:15:55.541989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.541906 5188 generic.go:356] "Generic (PLEG): container finished" podID="1b23775a-33a5-47b0-ac74-fb3862912542" containerID="f67ff330551f26f4b26b81619dee758936ee735b192639117f681f59708bc3a0" exitCode=255 Oct 31 21:15:55.541989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.541965 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" event={"ID":"1b23775a-33a5-47b0-ac74-fb3862912542","Type":"ContainerDied","Data":"f67ff330551f26f4b26b81619dee758936ee735b192639117f681f59708bc3a0"} Oct 31 21:15:55.542114 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.542007 5188 scope.go:122] "RemoveContainer" containerID="0728d08f5fd76e870a74999860ecc1ce866ab9547dabb2fedb55a89978d2fab1" Oct 31 21:15:55.542286 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.542252 5188 scope.go:122] "RemoveContainer" containerID="f67ff330551f26f4b26b81619dee758936ee735b192639117f681f59708bc3a0" Oct 31 21:15:55.542536 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:55.542507 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cg8tz_openshift-console-operator(1b23775a-33a5-47b0-ac74-fb3862912542)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" podUID="1b23775a-33a5-47b0-ac74-fb3862912542" Oct 31 21:15:55.544115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.544092 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c58487768-swgvs" event={"ID":"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b","Type":"ContainerStarted","Data":"65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de"} Oct 31 21:15:55.544202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.544131 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c58487768-swgvs" event={"ID":"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b","Type":"ContainerStarted","Data":"1b0c3a5f660b11252aa59c03ebc12b2d7d3d8e93020ac73fbfe298db76dfc2b3"} Oct 31 21:15:55.773005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.772950 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:55.776514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:55.776483 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:56.206223 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:56.206311 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:56.206369 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:56.206382 5188 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:56.206458 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:15:58.206441995 +0000 UTC m=+4.899399379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : secret "metrics-daemon-secret" not found Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:56.206474 5188 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:56.206505 5188 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:56.206534 5188 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:56.206478 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:56.206534 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls podName:c01f3957-4a49-41ba-812b-ea2e74d3a7a8 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:58.206518288 +0000 UTC m=+4.899475666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8r9vm" (UID: "c01f3957-4a49-41ba-812b-ea2e74d3a7a8") : secret "cluster-monitoring-operator-tls" not found Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:56.206603 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert podName:0bd131db-e6d0-46c6-9a4b-85d3b541a9ec nodeName:}" failed. No retries permitted until 2026-10-31 21:15:58.206588973 +0000 UTC m=+4.899546351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sh7vx" (UID: "0bd131db-e6d0-46c6-9a4b-85d3b541a9ec") : secret "networking-console-plugin-cert" not found Oct 31 21:15:56.206641 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:56.206621 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:15:58.206610195 +0000 UTC m=+4.899567572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : secret "insights-runtime-extractor-tls" not found Oct 31 21:15:56.553374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:56.553266 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/2.log" Oct 31 21:15:56.553775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:56.553661 5188 scope.go:122] "RemoveContainer" containerID="f67ff330551f26f4b26b81619dee758936ee735b192639117f681f59708bc3a0" Oct 31 21:15:56.553911 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:56.553880 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cg8tz_openshift-console-operator(1b23775a-33a5-47b0-ac74-fb3862912542)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" podUID="1b23775a-33a5-47b0-ac74-fb3862912542" Oct 31 21:15:56.554216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:56.554194 5188 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 21:15:56.554421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:56.554396 5188 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 21:15:57.558394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:57.558353 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" event={"ID":"e49a2d50-dc82-4983-9425-053dfe903ddd","Type":"ContainerStarted","Data":"38a99dd4bd7ba67dfbdf3e8cf4c1535c7bf2ca6ebfd13a1671edf4958bdd0fb2"} Oct 31 21:15:57.558886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:57.558424 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" event={"ID":"e49a2d50-dc82-4983-9425-053dfe903ddd","Type":"ContainerStarted","Data":"82cefe810f98c18337fe0482d7e6506abefe0a7091297bc9899cf5101a3a6dce"} Oct 31 21:15:57.560175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:57.560153 5188 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 21:15:57.560175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:57.560160 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wj49j" event={"ID":"b6f98741-1d48-4124-9acb-8c0bf31dbe60","Type":"ContainerStarted","Data":"f19913b7bb5fe66203af4dc8f041a6b4dda66f75b00a474a677a719785865e1e"} Oct 31 21:15:57.560359 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:57.560208 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wj49j" event={"ID":"b6f98741-1d48-4124-9acb-8c0bf31dbe60","Type":"ContainerStarted","Data":"a2e3e89b95c04dd4ddfd0f8f9768d7c2f69fd3d5a88dc9f35bed3ab2d9ef6486"} Oct 31 21:15:57.560395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:57.560355 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wj49j" Oct 31 21:15:58.230503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.230462 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:58.230503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.230507 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:58.230725 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.230564 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:15:58.230725 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.230601 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:15:58.230725 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:58.230690 5188 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 21:15:58.230825 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:58.230757 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs podName:3616b3de-b6e6-497f-8546-8fef766c309f nodeName:}" failed. No retries permitted until 2026-10-31 21:16:02.230738979 +0000 UTC m=+8.923696367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs") pod "network-metrics-daemon-zc54c" (UID: "3616b3de-b6e6-497f-8546-8fef766c309f") : secret "metrics-daemon-secret" not found Oct 31 21:15:58.230825 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:58.230690 5188 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 21:15:58.230924 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:58.230910 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls podName:7245fb80-34d5-471b-984e-7fbfe2b1113c nodeName:}" failed. No retries permitted until 2026-10-31 21:16:02.230886783 +0000 UTC m=+8.923844168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhxqp" (UID: "7245fb80-34d5-471b-984e-7fbfe2b1113c") : secret "insights-runtime-extractor-tls" not found Oct 31 21:15:58.232991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.232963 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c01f3957-4a49-41ba-812b-ea2e74d3a7a8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8r9vm\" (UID: \"c01f3957-4a49-41ba-812b-ea2e74d3a7a8\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:58.233126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.233102 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0bd131db-e6d0-46c6-9a4b-85d3b541a9ec-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sh7vx\" (UID: \"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:58.374638 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.374595 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" Oct 31 21:15:58.375660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.375633 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" Oct 31 21:15:58.566495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.566464 5188 generic.go:356] "Generic (PLEG): container finished" podID="31682289-5cba-480d-88a4-c71e9cd6244b" containerID="e285ee5c03acfccd850fce84974166a73e70c55a8c44d1fb22ad2861554f963c" exitCode=0 Oct 31 21:15:58.566958 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.566537 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" event={"ID":"31682289-5cba-480d-88a4-c71e9cd6244b","Type":"ContainerDied","Data":"e285ee5c03acfccd850fce84974166a73e70c55a8c44d1fb22ad2861554f963c"} Oct 31 21:15:58.567114 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.567097 5188 scope.go:122] "RemoveContainer" containerID="e285ee5c03acfccd850fce84974166a73e70c55a8c44d1fb22ad2861554f963c" Oct 31 21:15:58.648225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.648140 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:15:58.648641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.648624 5188 scope.go:122] "RemoveContainer" containerID="f67ff330551f26f4b26b81619dee758936ee735b192639117f681f59708bc3a0" Oct 31 21:15:58.648871 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:15:58.648850 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cg8tz_openshift-console-operator(1b23775a-33a5-47b0-ac74-fb3862912542)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" podUID="1b23775a-33a5-47b0-ac74-fb3862912542" Oct 31 21:15:58.680132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.680098 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm"] Oct 31 21:15:58.683133 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:58.683097 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01f3957_4a49_41ba_812b_ea2e74d3a7a8.slice/crio-16ee6b6ba7ba9bc6da1084f557065ef90bd6b3604381632b14eb2fa8d16aa096 WatchSource:0}: Error finding container 16ee6b6ba7ba9bc6da1084f557065ef90bd6b3604381632b14eb2fa8d16aa096: Status 404 returned error can't find the container with id 16ee6b6ba7ba9bc6da1084f557065ef90bd6b3604381632b14eb2fa8d16aa096 Oct 31 21:15:58.690772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:58.690747 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sh7vx"] Oct 31 21:15:58.693450 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:15:58.693429 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd131db_e6d0_46c6_9a4b_85d3b541a9ec.slice/crio-ddf4168104c91680ca8abf57634ab7fcec0f67e522ab587cbbb5b9ad77ee7d55 WatchSource:0}: Error finding container ddf4168104c91680ca8abf57634ab7fcec0f67e522ab587cbbb5b9ad77ee7d55: Status 404 returned error can't find the container with id ddf4168104c91680ca8abf57634ab7fcec0f67e522ab587cbbb5b9ad77ee7d55 Oct 31 21:15:59.000467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:59.000388 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:15:59.022980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:59.022947 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:15:59.172834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:59.172796 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:15:59.578255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:59.578205 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" event={"ID":"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec","Type":"ContainerStarted","Data":"ddf4168104c91680ca8abf57634ab7fcec0f67e522ab587cbbb5b9ad77ee7d55"} Oct 31 21:15:59.579821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:59.579761 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" event={"ID":"c01f3957-4a49-41ba-812b-ea2e74d3a7a8","Type":"ContainerStarted","Data":"16ee6b6ba7ba9bc6da1084f557065ef90bd6b3604381632b14eb2fa8d16aa096"} Oct 31 21:15:59.582096 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:15:59.582062 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p7ctk" event={"ID":"31682289-5cba-480d-88a4-c71e9cd6244b","Type":"ContainerStarted","Data":"077dfed73ddcf594b76365d32063ade3936f80c9ea20ed415512e1202bb9df7c"} Oct 31 21:16:00.360947 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:00.360894 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:16:00.361139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:00.361063 5188 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 21:16:00.362420 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:00.362396 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-68d4c65fcc-47wdn" Oct 31 21:16:00.381707 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:00.381668 5188 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kbsn_ac6429fb-48a7-4691-9079-4ed168f1da32/konnectivity-agent/0.log" limit=1024 Oct 31 21:16:00.538224 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:00.538186 5188 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kbsn_ac6429fb-48a7-4691-9079-4ed168f1da32/konnectivity-agent/0.log" limit=1024 Oct 31 21:16:00.588550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:00.588511 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sh7vx" event={"ID":"0bd131db-e6d0-46c6-9a4b-85d3b541a9ec","Type":"ContainerStarted","Data":"9bd91eed35f30c5f921b8078e622658d55bfd405d8473e625c735fa2725e5c00"} Oct 31 21:16:00.724928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:00.724889 5188 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kbsn_ac6429fb-48a7-4691-9079-4ed168f1da32/konnectivity-agent/0.log" limit=1024 Oct 31 21:16:00.921526 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:00.921412 5188 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kbsn_ac6429fb-48a7-4691-9079-4ed168f1da32/konnectivity-agent/0.log" limit=1024 Oct 31 21:16:01.120848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:01.120811 5188 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9kbsn_ac6429fb-48a7-4691-9079-4ed168f1da32/konnectivity-agent/0.log" limit=1024 Oct 31 21:16:01.596770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:01.596728 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8r9vm" event={"ID":"c01f3957-4a49-41ba-812b-ea2e74d3a7a8","Type":"ContainerStarted","Data":"acf77700ff71a189c05309a902c5f65129436d2e29e4940da2fe4cf256e7876a"} Oct 31 21:16:02.273077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.273023 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:16:02.273077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.273081 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:16:02.275581 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.275545 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7245fb80-34d5-471b-984e-7fbfe2b1113c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhxqp\" (UID: \"7245fb80-34d5-471b-984e-7fbfe2b1113c\") " pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:16:02.275714 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.275584 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3616b3de-b6e6-497f-8546-8fef766c309f-metrics-certs\") pod \"network-metrics-daemon-zc54c\" (UID: \"3616b3de-b6e6-497f-8546-8fef766c309f\") " pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:16:02.279519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.279482 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zc54c" Oct 31 21:16:02.289782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.289756 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dhxqp" Oct 31 21:16:02.581727 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.581648 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zc54c"] Oct 31 21:16:02.585036 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:16:02.585000 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3616b3de_b6e6_497f_8546_8fef766c309f.slice/crio-0f9f9cb21c91c47783e9739cca123fbd184a4d9582deeaf2148e2fa18ded8192 WatchSource:0}: Error finding container 0f9f9cb21c91c47783e9739cca123fbd184a4d9582deeaf2148e2fa18ded8192: Status 404 returned error can't find the container with id 0f9f9cb21c91c47783e9739cca123fbd184a4d9582deeaf2148e2fa18ded8192 Oct 31 21:16:02.595424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.595394 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dhxqp"] Oct 31 21:16:02.598511 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:16:02.598481 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7245fb80_34d5_471b_984e_7fbfe2b1113c.slice/crio-18505455333dcafc0a6a11fd01e3d87fdde808b62b803edfb635c35ac38980e6 WatchSource:0}: Error finding container 18505455333dcafc0a6a11fd01e3d87fdde808b62b803edfb635c35ac38980e6: Status 404 returned error can't find the container with id 18505455333dcafc0a6a11fd01e3d87fdde808b62b803edfb635c35ac38980e6 Oct 31 21:16:02.604872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.604834 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zc54c" event={"ID":"3616b3de-b6e6-497f-8546-8fef766c309f","Type":"ContainerStarted","Data":"0f9f9cb21c91c47783e9739cca123fbd184a4d9582deeaf2148e2fa18ded8192"} Oct 31 21:16:02.898896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:02.898863 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9kbsn" Oct 31 21:16:03.610383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:03.610353 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zc54c" event={"ID":"3616b3de-b6e6-497f-8546-8fef766c309f","Type":"ContainerStarted","Data":"7abef86260e07676ac7d0419fe8cd9281fd49a3fa0e211fb5caf2dd601233695"} Oct 31 21:16:03.612264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:03.612237 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhxqp" event={"ID":"7245fb80-34d5-471b-984e-7fbfe2b1113c","Type":"ContainerStarted","Data":"125c49c57f5a0cc2564cbe0dd3305c81ba8dec83110ae04d0484be97fc515a7f"} Oct 31 21:16:03.612379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:03.612290 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhxqp" event={"ID":"7245fb80-34d5-471b-984e-7fbfe2b1113c","Type":"ContainerStarted","Data":"a2237817840a495f1d87d8a185ddc5f847019e789024e1e0723416a232f3c71d"} Oct 31 21:16:03.612379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:03.612309 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhxqp" event={"ID":"7245fb80-34d5-471b-984e-7fbfe2b1113c","Type":"ContainerStarted","Data":"18505455333dcafc0a6a11fd01e3d87fdde808b62b803edfb635c35ac38980e6"} Oct 31 21:16:03.749541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:03.749504 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:16:04.335313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:04.335281 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:16:04.619842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:04.619804 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zc54c" event={"ID":"3616b3de-b6e6-497f-8546-8fef766c309f","Type":"ContainerStarted","Data":"7373cc4a04e6ae90c87793b79a1f87e9599c8de8078411603c34bd4264e5bffd"} Oct 31 21:16:04.622411 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:04.622384 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhxqp" event={"ID":"7245fb80-34d5-471b-984e-7fbfe2b1113c","Type":"ContainerStarted","Data":"95cf22679968d5e07731c1e15ee828a52ac2fc29ff4b33fd836a200a6d5ffa63"} Oct 31 21:16:04.622519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:04.622417 5188 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 21:16:05.051500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:05.051419 5188 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:16:05.051832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:05.051817 5188 scope.go:122] "RemoveContainer" containerID="f67ff330551f26f4b26b81619dee758936ee735b192639117f681f59708bc3a0" Oct 31 21:16:05.631439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:05.631383 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:16:05.631853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:05.631811 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/2.log" Oct 31 21:16:05.631853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:05.631843 5188 generic.go:356] "Generic (PLEG): container finished" podID="1b23775a-33a5-47b0-ac74-fb3862912542" containerID="6ec6f4a037ed8b0cbcb23e2f5dd3f5d25b95b0fe8cdcfd2cb9579785c4614161" exitCode=255 Oct 31 21:16:05.631947 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:05.631923 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" event={"ID":"1b23775a-33a5-47b0-ac74-fb3862912542","Type":"ContainerDied","Data":"6ec6f4a037ed8b0cbcb23e2f5dd3f5d25b95b0fe8cdcfd2cb9579785c4614161"} Oct 31 21:16:05.632015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:05.631988 5188 scope.go:122] "RemoveContainer" containerID="f67ff330551f26f4b26b81619dee758936ee735b192639117f681f59708bc3a0" Oct 31 21:16:05.632522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:05.632502 5188 scope.go:122] "RemoveContainer" containerID="6ec6f4a037ed8b0cbcb23e2f5dd3f5d25b95b0fe8cdcfd2cb9579785c4614161" Oct 31 21:16:05.632741 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:16:05.632711 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cg8tz_openshift-console-operator(1b23775a-33a5-47b0-ac74-fb3862912542)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" podUID="1b23775a-33a5-47b0-ac74-fb3862912542" Oct 31 21:16:06.637846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:06.637821 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:16:07.569255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:07.569214 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wj49j" Oct 31 21:16:08.648474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:08.648438 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:16:08.648888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:08.648816 5188 scope.go:122] "RemoveContainer" containerID="6ec6f4a037ed8b0cbcb23e2f5dd3f5d25b95b0fe8cdcfd2cb9579785c4614161" Oct 31 21:16:08.649029 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:16:08.649010 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cg8tz_openshift-console-operator(1b23775a-33a5-47b0-ac74-fb3862912542)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" podUID="1b23775a-33a5-47b0-ac74-fb3862912542" Oct 31 21:16:09.736606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:09.736569 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-574b9c7469-m4qfg"] Oct 31 21:16:11.856851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:11.856809 5188 patch_prober.go:29] interesting pod/image-registry-c58487768-swgvs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 21:16:11.857284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:11.856869 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c58487768-swgvs" podUID="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:16:15.051567 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:15.051508 5188 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:16:15.052005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:15.051991 5188 scope.go:122] "RemoveContainer" containerID="6ec6f4a037ed8b0cbcb23e2f5dd3f5d25b95b0fe8cdcfd2cb9579785c4614161" Oct 31 21:16:15.052202 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:16:15.052180 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cg8tz_openshift-console-operator(1b23775a-33a5-47b0-ac74-fb3862912542)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" podUID="1b23775a-33a5-47b0-ac74-fb3862912542" Oct 31 21:16:16.951267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:16.951236 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8nl7t" Oct 31 21:16:18.575174 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.575138 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7hdr5"] Oct 31 21:16:18.607830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.607796 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.634908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.634863 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 21:16:18.634908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.634864 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xn5d6\"" Oct 31 21:16:18.634908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.634902 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 21:16:18.644452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.644422 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 21:16:18.644452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.644429 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 21:16:18.704500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.704453 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-textfile\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.704500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.704507 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.704729 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.704584 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-wtmp\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.704729 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.704654 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d774b4f-526b-4b95-9620-7bfa56cb12a6-metrics-client-ca\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.704729 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.704684 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d774b4f-526b-4b95-9620-7bfa56cb12a6-sys\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.704822 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.704757 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-tls\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.704822 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.704796 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjv6x\" (UniqueName: \"kubernetes.io/projected/4d774b4f-526b-4b95-9620-7bfa56cb12a6-kube-api-access-qjv6x\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.704917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.704824 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.704976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.704936 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d774b4f-526b-4b95-9620-7bfa56cb12a6-root\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.805606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805565 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjv6x\" (UniqueName: \"kubernetes.io/projected/4d774b4f-526b-4b95-9620-7bfa56cb12a6-kube-api-access-qjv6x\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.805807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805622 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.805807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805656 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d774b4f-526b-4b95-9620-7bfa56cb12a6-root\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.805807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805698 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-textfile\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.805807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805727 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.805807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805793 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d774b4f-526b-4b95-9620-7bfa56cb12a6-root\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.806071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805843 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-wtmp\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.806071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805893 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d774b4f-526b-4b95-9620-7bfa56cb12a6-metrics-client-ca\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.806071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805924 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d774b4f-526b-4b95-9620-7bfa56cb12a6-sys\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.806071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805952 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-wtmp\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.806071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805967 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-tls\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.806071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.805994 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d774b4f-526b-4b95-9620-7bfa56cb12a6-sys\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.806312 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.806146 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-textfile\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.806428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.806409 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-accelerators-collector-config\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.806428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.806420 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d774b4f-526b-4b95-9620-7bfa56cb12a6-metrics-client-ca\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.808300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.808273 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-tls\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.808434 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.808302 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d774b4f-526b-4b95-9620-7bfa56cb12a6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.826683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.826610 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjv6x\" (UniqueName: \"kubernetes.io/projected/4d774b4f-526b-4b95-9620-7bfa56cb12a6-kube-api-access-qjv6x\") pod \"node-exporter-7hdr5\" (UID: \"4d774b4f-526b-4b95-9620-7bfa56cb12a6\") " pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.958391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:18.958354 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7hdr5" Oct 31 21:16:18.999459 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:16:18.999420 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d774b4f_526b_4b95_9620_7bfa56cb12a6.slice/crio-6fd09ead91abde99ea8579ab5d3ec4c3b896d485072e907d92d66f6a2e25a57e WatchSource:0}: Error finding container 6fd09ead91abde99ea8579ab5d3ec4c3b896d485072e907d92d66f6a2e25a57e: Status 404 returned error can't find the container with id 6fd09ead91abde99ea8579ab5d3ec4c3b896d485072e907d92d66f6a2e25a57e Oct 31 21:16:19.178349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:19.178295 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:16:19.696037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:19.695995 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hdr5" event={"ID":"4d774b4f-526b-4b95-9620-7bfa56cb12a6","Type":"ContainerStarted","Data":"6fd09ead91abde99ea8579ab5d3ec4c3b896d485072e907d92d66f6a2e25a57e"} Oct 31 21:16:19.744106 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:19.744074 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:16:20.702389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:20.702316 5188 generic.go:356] "Generic (PLEG): container finished" podID="4d774b4f-526b-4b95-9620-7bfa56cb12a6" containerID="a32f2d3ddaf29cc680227aacf7b6488857e91e6b6a9cc288eafc8dbbcff64d53" exitCode=0 Oct 31 21:16:20.702837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:20.702404 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hdr5" event={"ID":"4d774b4f-526b-4b95-9620-7bfa56cb12a6","Type":"ContainerDied","Data":"a32f2d3ddaf29cc680227aacf7b6488857e91e6b6a9cc288eafc8dbbcff64d53"} Oct 31 21:16:21.708410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:21.708376 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hdr5" event={"ID":"4d774b4f-526b-4b95-9620-7bfa56cb12a6","Type":"ContainerStarted","Data":"87cbb0837adce47f656b58fee0a7f9bf5c05aa6c4536589bb730736029fffae8"} Oct 31 21:16:21.708932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:21.708429 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7hdr5" event={"ID":"4d774b4f-526b-4b95-9620-7bfa56cb12a6","Type":"ContainerStarted","Data":"eadd35c6de3b80ca8f56ab42d9c88052d20a32f25a7cc813cc9d19abc1b8dd4c"} Oct 31 21:16:21.767144 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:21.767087 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7hdr5" podStartSLOduration=2.327087335 podStartE2EDuration="3.767072279s" totalImagesPullingTime="1.35222502s" totalInitContainerRuntime="87.759924ms" isStatefulPod=true podCreationTimestamp="2026-10-31 21:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:21.757213486 +0000 UTC m=+28.450170882" watchObservedRunningTime="2026-10-31 21:16:21.767072279 +0000 UTC m=+28.460029663" Oct 31 21:16:27.465699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:27.465661 5188 scope.go:122] "RemoveContainer" containerID="6ec6f4a037ed8b0cbcb23e2f5dd3f5d25b95b0fe8cdcfd2cb9579785c4614161" Oct 31 21:16:27.735636 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:27.735544 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:16:27.735790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:27.735643 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" event={"ID":"1b23775a-33a5-47b0-ac74-fb3862912542","Type":"ContainerStarted","Data":"b0a908049e0de0b6fd13295b118cd1e91e7f4ea2479463c80fb3e0dae50e8144"} Oct 31 21:16:27.735976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:27.735950 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:16:27.938277 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:27.938243 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cg8tz" Oct 31 21:16:29.003337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:29.003293 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tpl7t" Oct 31 21:16:32.263397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:32.263358 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c58487768-swgvs"] Oct 31 21:16:32.538839 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:16:32.538753 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:16:34.763440 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:34.763307 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" podUID="ce941b3a-da84-4b86-944c-89f9f464a27f" containerName="registry" containerID="cri-o://d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2" gracePeriod=30 Oct 31 21:16:35.023490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.023407 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:16:35.143457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.143422 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration\") pod \"ce941b3a-da84-4b86-944c-89f9f464a27f\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " Oct 31 21:16:35.143673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.143485 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca\") pod \"ce941b3a-da84-4b86-944c-89f9f464a27f\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " Oct 31 21:16:35.143673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.143525 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") pod \"ce941b3a-da84-4b86-944c-89f9f464a27f\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " Oct 31 21:16:35.143673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.143572 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates\") pod \"ce941b3a-da84-4b86-944c-89f9f464a27f\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " Oct 31 21:16:35.143673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.143634 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token\") pod \"ce941b3a-da84-4b86-944c-89f9f464a27f\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " Oct 31 21:16:35.143890 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.143678 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets\") pod \"ce941b3a-da84-4b86-944c-89f9f464a27f\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " Oct 31 21:16:35.143890 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.143820 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd\") pod \"ce941b3a-da84-4b86-944c-89f9f464a27f\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " Oct 31 21:16:35.143890 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.143875 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted\") pod \"ce941b3a-da84-4b86-944c-89f9f464a27f\" (UID: \"ce941b3a-da84-4b86-944c-89f9f464a27f\") " Oct 31 21:16:35.144083 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.144057 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca" pod "ce941b3a-da84-4b86-944c-89f9f464a27f" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:16:35.144226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.144057 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates" pod "ce941b3a-da84-4b86-944c-89f9f464a27f" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:16:35.146787 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.146739 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets" pod "ce941b3a-da84-4b86-944c-89f9f464a27f" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:16:35.146918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.146784 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token" pod "ce941b3a-da84-4b86-944c-89f9f464a27f" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:16:35.146918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.146825 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls" pod "ce941b3a-da84-4b86-944c-89f9f464a27f" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:16:35.146918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.146840 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration" pod "ce941b3a-da84-4b86-944c-89f9f464a27f" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:16:35.147041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.146915 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd" pod "ce941b3a-da84-4b86-944c-89f9f464a27f" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f"). InnerVolumeSpecName "kube-api-access-k5vtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:16:35.152801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.152739 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted" pod "ce941b3a-da84-4b86-944c-89f9f464a27f" (UID: "ce941b3a-da84-4b86-944c-89f9f464a27f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:16:35.245424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.245385 5188 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-bound-sa-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:35.245424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.245419 5188 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-installation-pull-secrets\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:35.245424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.245431 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vtd\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-kube-api-access-k5vtd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:35.245666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.245441 5188 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ce941b3a-da84-4b86-944c-89f9f464a27f-ca-trust-extracted\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:35.245666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.245453 5188 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ce941b3a-da84-4b86-944c-89f9f464a27f-image-registry-private-configuration\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:35.245666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.245463 5188 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-trusted-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:35.245666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.245472 5188 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:35.245666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.245480 5188 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ce941b3a-da84-4b86-944c-89f9f464a27f-registry-certificates\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:35.771674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.771573 5188 generic.go:356] "Generic (PLEG): container finished" podID="ce941b3a-da84-4b86-944c-89f9f464a27f" containerID="d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2" exitCode=0 Oct 31 21:16:35.771674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.771619 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" Oct 31 21:16:35.772175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.771672 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" event={"ID":"ce941b3a-da84-4b86-944c-89f9f464a27f","Type":"ContainerDied","Data":"d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2"} Oct 31 21:16:35.772175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.771733 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-574b9c7469-m4qfg" event={"ID":"ce941b3a-da84-4b86-944c-89f9f464a27f","Type":"ContainerDied","Data":"6564c8f268b71431e88970540ea281ab75021339847e5664015204fbb4616722"} Oct 31 21:16:35.772175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.771762 5188 scope.go:122] "RemoveContainer" containerID="d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2" Oct 31 21:16:35.780152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.780129 5188 scope.go:122] "RemoveContainer" containerID="d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2" Oct 31 21:16:35.780510 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:16:35.780482 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2\": container with ID starting with d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2 not found: ID does not exist" containerID="d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2" Oct 31 21:16:35.780575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.780521 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2"} err="failed to get container status \"d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2\": rpc error: code = NotFound desc = could not find container \"d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2\": container with ID starting with d332a1b8707b00cc5740d9fa5601bd7cee11d3f9bead1572177e9117405228d2 not found: ID does not exist" Oct 31 21:16:35.825763 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.825721 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-574b9c7469-m4qfg"] Oct 31 21:16:35.835356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:35.835308 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-574b9c7469-m4qfg"] Oct 31 21:16:37.470477 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:37.470445 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce941b3a-da84-4b86-944c-89f9f464a27f" path="/var/lib/kubelet/pods/ce941b3a-da84-4b86-944c-89f9f464a27f/volumes" Oct 31 21:16:57.283478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.283431 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c58487768-swgvs" podUID="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" containerName="registry" containerID="cri-o://65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de" gracePeriod=30 Oct 31 21:16:57.530761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.530730 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:16:57.641160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641122 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets\") pod \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " Oct 31 21:16:57.641357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641175 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") pod \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " Oct 31 21:16:57.641357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641246 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates\") pod \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " Oct 31 21:16:57.641357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641281 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca\") pod \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " Oct 31 21:16:57.641357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641313 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted\") pod \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " Oct 31 21:16:57.641569 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641363 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration\") pod \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " Oct 31 21:16:57.641569 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641386 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token\") pod \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " Oct 31 21:16:57.641569 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641422 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg\") pod \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\" (UID: \"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b\") " Oct 31 21:16:57.641847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641815 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca" pod "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:16:57.641906 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.641854 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates" pod "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:16:57.644119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.644078 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token" pod "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:16:57.644230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.644079 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets" pod "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:16:57.644230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.644085 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls" pod "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:16:57.644230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.644157 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration" pod "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:16:57.644424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.644400 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg" pod "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b"). InnerVolumeSpecName "kube-api-access-j6qmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:16:57.648603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.648539 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted" pod "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" (UID: "5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:16:57.742640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.742602 5188 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-certificates\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:57.742640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.742633 5188 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-trusted-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:57.742640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.742644 5188 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-ca-trust-extracted\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:57.742882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.742655 5188 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-image-registry-private-configuration\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:57.742882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.742665 5188 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-bound-sa-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:57.742882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.742673 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6qmg\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-kube-api-access-j6qmg\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:57.742882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.742682 5188 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-installation-pull-secrets\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:57.742882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.742691 5188 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b-registry-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:16:57.858602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.858567 5188 generic.go:356] "Generic (PLEG): container finished" podID="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" containerID="65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de" exitCode=0 Oct 31 21:16:57.858785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.858621 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c58487768-swgvs" Oct 31 21:16:57.858785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.858653 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c58487768-swgvs" event={"ID":"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b","Type":"ContainerDied","Data":"65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de"} Oct 31 21:16:57.858785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.858709 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c58487768-swgvs" event={"ID":"5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b","Type":"ContainerDied","Data":"1b0c3a5f660b11252aa59c03ebc12b2d7d3d8e93020ac73fbfe298db76dfc2b3"} Oct 31 21:16:57.858785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.858737 5188 scope.go:122] "RemoveContainer" containerID="65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de" Oct 31 21:16:57.872126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.872106 5188 scope.go:122] "RemoveContainer" containerID="65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de" Oct 31 21:16:57.872462 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:16:57.872437 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de\": container with ID starting with 65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de not found: ID does not exist" containerID="65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de" Oct 31 21:16:57.872526 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.872472 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de"} err="failed to get container status \"65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de\": rpc error: code = NotFound desc = could not find container \"65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de\": container with ID starting with 65e4d7c5dc61fd51070a84a8877bca1079d987d09430b95ebe44ebc8ccec62de not found: ID does not exist" Oct 31 21:16:57.903009 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.902928 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c58487768-swgvs"] Oct 31 21:16:57.912757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:57.912725 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c58487768-swgvs"] Oct 31 21:16:59.470704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:16:59.470666 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" path="/var/lib/kubelet/pods/5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b/volumes" Oct 31 21:17:02.540509 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:17:02.540472 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:17:07.901605 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:07.901570 5188 generic.go:356] "Generic (PLEG): container finished" podID="d2a47228-9b40-4833-bb84-970f4782e67b" containerID="43fc9c1ba32e4402ba3e187e507f3f10a97462eefa6747b2a0ac6c7d0fd4254e" exitCode=0 Oct 31 21:17:07.902004 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:07.901648 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnbr9" event={"ID":"d2a47228-9b40-4833-bb84-970f4782e67b","Type":"ContainerDied","Data":"43fc9c1ba32e4402ba3e187e507f3f10a97462eefa6747b2a0ac6c7d0fd4254e"} Oct 31 21:17:07.902044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:07.902029 5188 scope.go:122] "RemoveContainer" containerID="43fc9c1ba32e4402ba3e187e507f3f10a97462eefa6747b2a0ac6c7d0fd4254e" Oct 31 21:17:08.405783 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.405743 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vk6tn-jprcv"] Oct 31 21:17:08.406055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.406014 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" podUID="29050b65-73ea-446c-98b6-76ce5438f0c9" containerName="insights-gathering" containerID="cri-o://6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51" gracePeriod=30 Oct 31 21:17:08.583078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.582690 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wwmjx-d8ctc"] Oct 31 21:17:08.583739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.583285 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" containerName="registry" Oct 31 21:17:08.583739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.583305 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" containerName="registry" Oct 31 21:17:08.583739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.583372 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ce941b3a-da84-4b86-944c-89f9f464a27f" containerName="registry" Oct 31 21:17:08.583739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.583382 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce941b3a-da84-4b86-944c-89f9f464a27f" containerName="registry" Oct 31 21:17:08.583739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.583529 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce941b3a-da84-4b86-944c-89f9f464a27f" containerName="registry" Oct 31 21:17:08.583739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.583546 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e27e3a3-5abc-4fe2-87f6-01dfccdb5d1b" containerName="registry" Oct 31 21:17:08.586881 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.586832 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.606619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.606246 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wwmjx-d8ctc"] Oct 31 21:17:08.624444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.624396 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ab8c2194-2d6c-4448-a90a-bc3079740049-archives-path\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.624612 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.624474 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-trusted-ca-bundle\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.624612 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.624560 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2597b\" (UniqueName: \"kubernetes.io/projected/ab8c2194-2d6c-4448-a90a-bc3079740049-kube-api-access-2597b\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.624683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.624608 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-service-ca-bundle\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.657773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.657707 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:17:08.725472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725428 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle\") pod \"29050b65-73ea-446c-98b6-76ce5438f0c9\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " Oct 31 21:17:08.725649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725513 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6\" (UniqueName: \"kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6\") pod \"29050b65-73ea-446c-98b6-76ce5438f0c9\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " Oct 31 21:17:08.725649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725556 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle\") pod \"29050b65-73ea-446c-98b6-76ce5438f0c9\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " Oct 31 21:17:08.725649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725593 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path\") pod \"29050b65-73ea-446c-98b6-76ce5438f0c9\" (UID: \"29050b65-73ea-446c-98b6-76ce5438f0c9\") " Oct 31 21:17:08.725829 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725696 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-service-ca-bundle\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.725829 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725737 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ab8c2194-2d6c-4448-a90a-bc3079740049-archives-path\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.725829 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725813 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-trusted-ca-bundle\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.725974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725886 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2597b\" (UniqueName: \"kubernetes.io/projected/ab8c2194-2d6c-4448-a90a-bc3079740049-kube-api-access-2597b\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.725974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725921 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle" pod "29050b65-73ea-446c-98b6-76ce5438f0c9" (UID: "29050b65-73ea-446c-98b6-76ce5438f0c9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:08.726067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.725969 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle" pod "29050b65-73ea-446c-98b6-76ce5438f0c9" (UID: "29050b65-73ea-446c-98b6-76ce5438f0c9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:08.726067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.726009 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path" pod "29050b65-73ea-446c-98b6-76ce5438f0c9" (UID: "29050b65-73ea-446c-98b6-76ce5438f0c9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:17:08.726243 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.726221 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ab8c2194-2d6c-4448-a90a-bc3079740049-archives-path\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.726586 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.726561 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-service-ca-bundle\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.726850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.726832 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-trusted-ca-bundle\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.728056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.728036 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6" pod "29050b65-73ea-446c-98b6-76ce5438f0c9" (UID: "29050b65-73ea-446c-98b6-76ce5438f0c9"). InnerVolumeSpecName "kube-api-access-7dnm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:17:08.747975 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.747936 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2597b\" (UniqueName: \"kubernetes.io/projected/ab8c2194-2d6c-4448-a90a-bc3079740049-kube-api-access-2597b\") pod \"periodic-gathering-wwmjx-d8ctc\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.826705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.826670 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dnm6\" (UniqueName: \"kubernetes.io/projected/29050b65-73ea-446c-98b6-76ce5438f0c9-kube-api-access-7dnm6\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:17:08.826705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.826701 5188 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-service-ca-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:17:08.826705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.826712 5188 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/29050b65-73ea-446c-98b6-76ce5438f0c9-archives-path\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:17:08.826929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.826722 5188 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/29050b65-73ea-446c-98b6-76ce5438f0c9-trusted-ca-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:17:08.903402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.903363 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:17:08.907734 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.907254 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-bnbr9" event={"ID":"d2a47228-9b40-4833-bb84-970f4782e67b","Type":"ContainerStarted","Data":"e04fa490ebe270edaedc6904aad3386f39ad9b7516fd33e0f9de894331c142cc"} Oct 31 21:17:08.908666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.908604 5188 generic.go:356] "Generic (PLEG): container finished" podID="29050b65-73ea-446c-98b6-76ce5438f0c9" containerID="6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51" exitCode=2 Oct 31 21:17:08.908666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.908647 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" event={"ID":"29050b65-73ea-446c-98b6-76ce5438f0c9","Type":"ContainerDied","Data":"6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51"} Oct 31 21:17:08.908666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.908650 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" Oct 31 21:17:08.908845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.908683 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-vk6tn-jprcv" event={"ID":"29050b65-73ea-446c-98b6-76ce5438f0c9","Type":"ContainerDied","Data":"7c0d5c17583db4e70c67c6119379ee93e013f8fc26f49b2ec07eb16f785787f0"} Oct 31 21:17:08.908845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.908703 5188 scope.go:122] "RemoveContainer" containerID="6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51" Oct 31 21:17:08.926022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.925999 5188 scope.go:122] "RemoveContainer" containerID="6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51" Oct 31 21:17:08.926461 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:17:08.926439 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51\": container with ID starting with 6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51 not found: ID does not exist" containerID="6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51" Oct 31 21:17:08.926534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.926474 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51"} err="failed to get container status \"6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51\": rpc error: code = NotFound desc = could not find container \"6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51\": container with ID starting with 6fa45d76127ab0099a1342cb463a024ef815ba492f7a24fe1a5de1b8bf52cb51 not found: ID does not exist" Oct 31 21:17:08.976933 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.976895 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-vk6tn-jprcv"] Oct 31 21:17:08.983911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:08.983875 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-vk6tn-jprcv"] Oct 31 21:17:09.206976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:09.206896 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wwmjx-d8ctc"] Oct 31 21:17:09.210198 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:17:09.210159 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8c2194_2d6c_4448_a90a_bc3079740049.slice/crio-5d799f9054382ccdc81f78ef7bb2c4b9c5dd2fc15267abb3a028d1607388e869 WatchSource:0}: Error finding container 5d799f9054382ccdc81f78ef7bb2c4b9c5dd2fc15267abb3a028d1607388e869: Status 404 returned error can't find the container with id 5d799f9054382ccdc81f78ef7bb2c4b9c5dd2fc15267abb3a028d1607388e869 Oct 31 21:17:09.470721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:09.470642 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29050b65-73ea-446c-98b6-76ce5438f0c9" path="/var/lib/kubelet/pods/29050b65-73ea-446c-98b6-76ce5438f0c9/volumes" Oct 31 21:17:09.915530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:09.915495 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" event={"ID":"ab8c2194-2d6c-4448-a90a-bc3079740049","Type":"ContainerStarted","Data":"b4d34930b2f4b2d618cae03bacbac14405473811acd8d7098fc758ea9cb7768c"} Oct 31 21:17:09.915937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:09.915544 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" event={"ID":"ab8c2194-2d6c-4448-a90a-bc3079740049","Type":"ContainerStarted","Data":"5d799f9054382ccdc81f78ef7bb2c4b9c5dd2fc15267abb3a028d1607388e869"} Oct 31 21:17:09.952681 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:09.952623 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" podStartSLOduration=1.95260919 podStartE2EDuration="1.95260919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:17:09.951577708 +0000 UTC m=+76.644535102" watchObservedRunningTime="2026-10-31 21:17:09.95260919 +0000 UTC m=+76.645566580" Oct 31 21:17:23.971517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:23.971476 5188 generic.go:356] "Generic (PLEG): container finished" podID="21a28634-04b5-4d1b-bf14-6bdc3b894f2d" containerID="e135dc16a77e1e00689eff2607c2eb2820145495534eba5724c5b72afe8815d6" exitCode=0 Oct 31 21:17:23.971944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:23.971549 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" event={"ID":"21a28634-04b5-4d1b-bf14-6bdc3b894f2d","Type":"ContainerDied","Data":"e135dc16a77e1e00689eff2607c2eb2820145495534eba5724c5b72afe8815d6"} Oct 31 21:17:23.971944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:23.971918 5188 scope.go:122] "RemoveContainer" containerID="e135dc16a77e1e00689eff2607c2eb2820145495534eba5724c5b72afe8815d6" Oct 31 21:17:24.977266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:17:24.977229 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t4jx6" event={"ID":"21a28634-04b5-4d1b-bf14-6bdc3b894f2d","Type":"ContainerStarted","Data":"82c69fa03d31f52dc83cdc68b05c15183a6cbbea9e7c3d5d6bea6942cd5ff955"} Oct 31 21:17:32.538117 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:17:32.538077 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:18:02.537358 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:18:02.537301 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:18:32.536562 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:18:32.536462 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:18:34.260510 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:34.260470 5188 generic.go:356] "Generic (PLEG): container finished" podID="ab8c2194-2d6c-4448-a90a-bc3079740049" containerID="b4d34930b2f4b2d618cae03bacbac14405473811acd8d7098fc758ea9cb7768c" exitCode=0 Oct 31 21:18:34.260911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:34.260544 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" event={"ID":"ab8c2194-2d6c-4448-a90a-bc3079740049","Type":"ContainerDied","Data":"b4d34930b2f4b2d618cae03bacbac14405473811acd8d7098fc758ea9cb7768c"} Oct 31 21:18:35.397784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.397751 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:18:35.406684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.406651 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab8c2194-2d6c-4448-a90a-bc3079740049-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ab8c2194-2d6c-4448-a90a-bc3079740049-archives-path\") pod \"ab8c2194-2d6c-4448-a90a-bc3079740049\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " Oct 31 21:18:35.406846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.406717 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab8c2194-2d6c-4448-a90a-bc3079740049-kube-api-access-2597b\" (UniqueName: \"kubernetes.io/projected/ab8c2194-2d6c-4448-a90a-bc3079740049-kube-api-access-2597b\") pod \"ab8c2194-2d6c-4448-a90a-bc3079740049\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " Oct 31 21:18:35.406846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.406770 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-trusted-ca-bundle\") pod \"ab8c2194-2d6c-4448-a90a-bc3079740049\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " Oct 31 21:18:35.406846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.406803 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-service-ca-bundle\") pod \"ab8c2194-2d6c-4448-a90a-bc3079740049\" (UID: \"ab8c2194-2d6c-4448-a90a-bc3079740049\") " Oct 31 21:18:35.407092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.407061 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8c2194-2d6c-4448-a90a-bc3079740049-archives-path" pod "ab8c2194-2d6c-4448-a90a-bc3079740049" (UID: "ab8c2194-2d6c-4448-a90a-bc3079740049"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:18:35.407163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.407142 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-trusted-ca-bundle" pod "ab8c2194-2d6c-4448-a90a-bc3079740049" (UID: "ab8c2194-2d6c-4448-a90a-bc3079740049"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:18:35.407199 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.407145 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-service-ca-bundle" pod "ab8c2194-2d6c-4448-a90a-bc3079740049" (UID: "ab8c2194-2d6c-4448-a90a-bc3079740049"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:18:35.408920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.408895 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8c2194-2d6c-4448-a90a-bc3079740049-kube-api-access-2597b" pod "ab8c2194-2d6c-4448-a90a-bc3079740049" (UID: "ab8c2194-2d6c-4448-a90a-bc3079740049"). InnerVolumeSpecName "kube-api-access-2597b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:18:35.508088 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.508054 5188 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ab8c2194-2d6c-4448-a90a-bc3079740049-archives-path\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:18:35.508088 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.508083 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2597b\" (UniqueName: \"kubernetes.io/projected/ab8c2194-2d6c-4448-a90a-bc3079740049-kube-api-access-2597b\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:18:35.508088 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.508093 5188 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-trusted-ca-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:18:35.508315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:35.508102 5188 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab8c2194-2d6c-4448-a90a-bc3079740049-service-ca-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:18:36.270170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:36.270133 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" event={"ID":"ab8c2194-2d6c-4448-a90a-bc3079740049","Type":"ContainerDied","Data":"5d799f9054382ccdc81f78ef7bb2c4b9c5dd2fc15267abb3a028d1607388e869"} Oct 31 21:18:36.270170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:36.270163 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wwmjx-d8ctc" Oct 31 21:18:36.270418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:18:36.270180 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d799f9054382ccdc81f78ef7bb2c4b9c5dd2fc15267abb3a028d1607388e869" Oct 31 21:19:02.537071 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:19:02.537030 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:19:32.536243 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:19:32.536198 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:20:02.536711 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:20:02.536661 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:20:29.402872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.402834 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v"] Oct 31 21:20:29.405393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.403135 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ab8c2194-2d6c-4448-a90a-bc3079740049" containerName="insights-gathering" Oct 31 21:20:29.405393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.403146 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab8c2194-2d6c-4448-a90a-bc3079740049" containerName="insights-gathering" Oct 31 21:20:29.405393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.403168 5188 cpu_manager.go:414] "containerMap: removing container" podUID="29050b65-73ea-446c-98b6-76ce5438f0c9" containerName="insights-gathering" Oct 31 21:20:29.405393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.403173 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29050b65-73ea-446c-98b6-76ce5438f0c9" containerName="insights-gathering" Oct 31 21:20:29.405393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.403246 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab8c2194-2d6c-4448-a90a-bc3079740049" containerName="insights-gathering" Oct 31 21:20:29.405393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.403258 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="29050b65-73ea-446c-98b6-76ce5438f0c9" containerName="insights-gathering" Oct 31 21:20:29.406336 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.406306 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.419661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.419643 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 21:20:29.429607 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.429575 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 21:20:29.429726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.429638 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8vzj6\"" Oct 31 21:20:29.430772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.430748 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v"] Oct 31 21:20:29.524049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.524008 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.524230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.524056 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vc9g\" (UniqueName: \"kubernetes.io/projected/f7b2a240-2b0f-49b8-b529-c801c62c1160-kube-api-access-2vc9g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.524230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.524180 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.625343 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.625276 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.625542 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.625374 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.625542 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.625435 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vc9g\" (UniqueName: \"kubernetes.io/projected/f7b2a240-2b0f-49b8-b529-c801c62c1160-kube-api-access-2vc9g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.625723 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.625680 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.625723 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.625694 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.646843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.646808 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vc9g\" (UniqueName: \"kubernetes.io/projected/f7b2a240-2b0f-49b8-b529-c801c62c1160-kube-api-access-2vc9g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:29.722898 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:29.722807 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:20:30.024107 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:30.024038 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v"] Oct 31 21:20:30.027054 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:20:30.027022 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b2a240_2b0f_49b8_b529_c801c62c1160.slice/crio-a854288c77efec377effb5a656f9220df2dd5e43748a8d637ee9122850e15f4f WatchSource:0}: Error finding container a854288c77efec377effb5a656f9220df2dd5e43748a8d637ee9122850e15f4f: Status 404 returned error can't find the container with id a854288c77efec377effb5a656f9220df2dd5e43748a8d637ee9122850e15f4f Oct 31 21:20:30.713181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:30.713140 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" event={"ID":"f7b2a240-2b0f-49b8-b529-c801c62c1160","Type":"ContainerStarted","Data":"a854288c77efec377effb5a656f9220df2dd5e43748a8d637ee9122850e15f4f"} Oct 31 21:20:32.070266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.070230 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb"] Oct 31 21:20:32.074405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.074377 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.098267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.098241 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-c4wt7\"" Oct 31 21:20:32.099879 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.099852 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb"] Oct 31 21:20:32.173933 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.172304 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl"] Oct 31 21:20:32.176231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.176199 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.193576 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.193550 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-wg2m5\"" Oct 31 21:20:32.211897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.211862 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl"] Oct 31 21:20:32.250220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.250179 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.250220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.250223 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zr9q\" (UniqueName: \"kubernetes.io/projected/71cf5160-6ca6-45c7-9013-010caa843b72-kube-api-access-2zr9q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.250557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.250403 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.288491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.288454 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5"] Oct 31 21:20:32.292228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.292203 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.304890 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.304870 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-qxnmr\"" Oct 31 21:20:32.312275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.312248 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5"] Oct 31 21:20:32.350867 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.350784 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.350867 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.350846 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjpr\" (UniqueName: \"kubernetes.io/projected/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-kube-api-access-qtjpr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.351092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.350945 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.351092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.350987 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.351207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.351152 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.351207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.351190 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zr9q\" (UniqueName: \"kubernetes.io/projected/71cf5160-6ca6-45c7-9013-010caa843b72-kube-api-access-2zr9q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.351432 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.351412 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.351704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.351687 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.374127 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.374098 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zr9q\" (UniqueName: \"kubernetes.io/projected/71cf5160-6ca6-45c7-9013-010caa843b72-kube-api-access-2zr9q\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.397397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.397366 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:20:32.418395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.418349 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d"] Oct 31 21:20:32.422389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.422365 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.436053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.436024 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ktwx7\"" Oct 31 21:20:32.446024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.445987 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d"] Oct 31 21:20:32.451854 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.451821 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.452012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.451906 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.452012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.451948 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjpr\" (UniqueName: \"kubernetes.io/projected/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-kube-api-access-qtjpr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.452012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.451990 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.452193 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.452032 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdstl\" (UniqueName: \"kubernetes.io/projected/e25b4385-15fc-4fea-8099-977952f32171-kube-api-access-wdstl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.452193 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.452094 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.452539 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.452514 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.453023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.453001 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.497717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.497681 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjpr\" (UniqueName: \"kubernetes.io/projected/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-kube-api-access-qtjpr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.538909 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:20:32.538866 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:20:32.553437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.553399 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdstl\" (UniqueName: \"kubernetes.io/projected/e25b4385-15fc-4fea-8099-977952f32171-kube-api-access-wdstl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.553603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.553461 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.553603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.553484 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.553603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.553519 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkk94\" (UniqueName: \"kubernetes.io/projected/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-kube-api-access-rkk94\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.553603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.553553 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.553603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.553577 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.554029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.554005 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.554579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.554561 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.578633 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.578603 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdstl\" (UniqueName: \"kubernetes.io/projected/e25b4385-15fc-4fea-8099-977952f32171-kube-api-access-wdstl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.610535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.610505 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:32.655467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.655409 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.655786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.655476 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.655786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.655522 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkk94\" (UniqueName: \"kubernetes.io/projected/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-kube-api-access-rkk94\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.656455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.656426 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.656631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.656615 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.679188 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.679134 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkk94\" (UniqueName: \"kubernetes.io/projected/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-kube-api-access-rkk94\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.706560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.706533 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb"] Oct 31 21:20:32.709437 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:20:32.709398 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cf5160_6ca6_45c7_9013_010caa843b72.slice/crio-327ef882c9358caf987634d0b04ceae194492b7be68ca52465bb58253da2ee27 WatchSource:0}: Error finding container 327ef882c9358caf987634d0b04ceae194492b7be68ca52465bb58253da2ee27: Status 404 returned error can't find the container with id 327ef882c9358caf987634d0b04ceae194492b7be68ca52465bb58253da2ee27 Oct 31 21:20:32.724511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.724478 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" event={"ID":"71cf5160-6ca6-45c7-9013-010caa843b72","Type":"ContainerStarted","Data":"327ef882c9358caf987634d0b04ceae194492b7be68ca52465bb58253da2ee27"} Oct 31 21:20:32.745970 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.745938 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:20:32.794845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.794811 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:20:32.937519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:32.937488 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5"] Oct 31 21:20:32.939870 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:20:32.939831 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25b4385_15fc_4fea_8099_977952f32171.slice/crio-72240cfb5eb92f8b3f5297e280770e965a7d9c44aefee4ce5ddb1f01d08a7ae1 WatchSource:0}: Error finding container 72240cfb5eb92f8b3f5297e280770e965a7d9c44aefee4ce5ddb1f01d08a7ae1: Status 404 returned error can't find the container with id 72240cfb5eb92f8b3f5297e280770e965a7d9c44aefee4ce5ddb1f01d08a7ae1 Oct 31 21:20:33.070053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.070015 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d"] Oct 31 21:20:33.073433 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:20:33.073403 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3b462a_0202_47e2_a2f8_8d061f07ab8c.slice/crio-da39aa4f7d9c45c19fd0431256568b98fd92db2cd0ba97b346d2153288d48777 WatchSource:0}: Error finding container da39aa4f7d9c45c19fd0431256568b98fd92db2cd0ba97b346d2153288d48777: Status 404 returned error can't find the container with id da39aa4f7d9c45c19fd0431256568b98fd92db2cd0ba97b346d2153288d48777 Oct 31 21:20:33.110829 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.110805 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl"] Oct 31 21:20:33.112407 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:20:33.112341 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8bf7ad_6cc8_4838_952a_a6d13b57f5fc.slice/crio-a48bedbe0004f163cb7ea0f3edd74ae9dcf078aab0529b710c8c5e4ccf7316d5 WatchSource:0}: Error finding container a48bedbe0004f163cb7ea0f3edd74ae9dcf078aab0529b710c8c5e4ccf7316d5: Status 404 returned error can't find the container with id a48bedbe0004f163cb7ea0f3edd74ae9dcf078aab0529b710c8c5e4ccf7316d5 Oct 31 21:20:33.159083 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.159032 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh"] Oct 31 21:20:33.163543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.163124 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.180252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.180209 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-gkgc4\"" Oct 31 21:20:33.185929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.185902 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh"] Oct 31 21:20:33.260751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.260698 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85n9s\" (UniqueName: \"kubernetes.io/projected/613dec8c-fe05-4ddc-8a74-75efcba90c90-kube-api-access-85n9s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.260947 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.260759 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.260947 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.260820 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.361653 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.361612 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.361799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.361688 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85n9s\" (UniqueName: \"kubernetes.io/projected/613dec8c-fe05-4ddc-8a74-75efcba90c90-kube-api-access-85n9s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.361799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.361726 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.362160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.362135 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.362455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.362400 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.396362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.395051 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85n9s\" (UniqueName: \"kubernetes.io/projected/613dec8c-fe05-4ddc-8a74-75efcba90c90-kube-api-access-85n9s\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.483213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.482953 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:33.583577 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.583541 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx"] Oct 31 21:20:33.588357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.588044 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.606409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.606380 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9hcw6\"" Oct 31 21:20:33.608235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.608210 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx"] Oct 31 21:20:33.731947 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.731810 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" event={"ID":"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc","Type":"ContainerStarted","Data":"a48bedbe0004f163cb7ea0f3edd74ae9dcf078aab0529b710c8c5e4ccf7316d5"} Oct 31 21:20:33.734474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.734392 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" event={"ID":"e25b4385-15fc-4fea-8099-977952f32171","Type":"ContainerStarted","Data":"72240cfb5eb92f8b3f5297e280770e965a7d9c44aefee4ce5ddb1f01d08a7ae1"} Oct 31 21:20:33.736198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.736150 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" event={"ID":"ce3b462a-0202-47e2-a2f8-8d061f07ab8c","Type":"ContainerStarted","Data":"da39aa4f7d9c45c19fd0431256568b98fd92db2cd0ba97b346d2153288d48777"} Oct 31 21:20:33.764895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.764597 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.764895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.764661 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7x6x\" (UniqueName: \"kubernetes.io/projected/7f50b1dc-158e-47de-99d3-557e9b2ee04c-kube-api-access-k7x6x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.764895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.764736 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.812423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.812027 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh"] Oct 31 21:20:33.866648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.865339 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.866648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.865460 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.866648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.865508 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7x6x\" (UniqueName: \"kubernetes.io/projected/7f50b1dc-158e-47de-99d3-557e9b2ee04c-kube-api-access-k7x6x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.866648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.866261 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.866648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.866590 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.902980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.902920 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7x6x\" (UniqueName: \"kubernetes.io/projected/7f50b1dc-158e-47de-99d3-557e9b2ee04c-kube-api-access-k7x6x\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:33.917499 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:33.917086 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:34.243603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:34.243311 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx"] Oct 31 21:20:34.247934 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:20:34.247884 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f50b1dc_158e_47de_99d3_557e9b2ee04c.slice/crio-072dc5b74d61562be3a0c4aaec715e9b8ed575b5e1e529616741d5beff0df539 WatchSource:0}: Error finding container 072dc5b74d61562be3a0c4aaec715e9b8ed575b5e1e529616741d5beff0df539: Status 404 returned error can't find the container with id 072dc5b74d61562be3a0c4aaec715e9b8ed575b5e1e529616741d5beff0df539 Oct 31 21:20:34.761201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:34.761154 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" event={"ID":"7f50b1dc-158e-47de-99d3-557e9b2ee04c","Type":"ContainerStarted","Data":"072dc5b74d61562be3a0c4aaec715e9b8ed575b5e1e529616741d5beff0df539"} Oct 31 21:20:34.764538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:34.764444 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" event={"ID":"613dec8c-fe05-4ddc-8a74-75efcba90c90","Type":"ContainerStarted","Data":"2aa472403456e4c660a61223795bed9cee73ed22ac39f7c4421cbb3f1a21cd1a"} Oct 31 21:20:39.812039 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:39.811959 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" event={"ID":"f7b2a240-2b0f-49b8-b529-c801c62c1160","Type":"ContainerStarted","Data":"f44c1590af175554174e270442268738511f29fb2d4c11f766426f5e51682de7"} Oct 31 21:20:40.822619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:40.822575 5188 generic.go:356] "Generic (PLEG): container finished" podID="e25b4385-15fc-4fea-8099-977952f32171" containerID="6c150cc8c929809e040fe34a5002fde9f234934c65583264e41bead60f19aa6e" exitCode=0 Oct 31 21:20:40.823133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:40.822558 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" event={"ID":"e25b4385-15fc-4fea-8099-977952f32171","Type":"ContainerDied","Data":"6c150cc8c929809e040fe34a5002fde9f234934c65583264e41bead60f19aa6e"} Oct 31 21:20:40.828423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:40.827618 5188 generic.go:356] "Generic (PLEG): container finished" podID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerID="f44c1590af175554174e270442268738511f29fb2d4c11f766426f5e51682de7" exitCode=0 Oct 31 21:20:40.828423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:40.827701 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" event={"ID":"f7b2a240-2b0f-49b8-b529-c801c62c1160","Type":"ContainerDied","Data":"f44c1590af175554174e270442268738511f29fb2d4c11f766426f5e51682de7"} Oct 31 21:20:41.835661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.835626 5188 generic.go:356] "Generic (PLEG): container finished" podID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerID="b6427ec34b228b961890815a999b7293dae4a87551e666827b5245f242cd672d" exitCode=0 Oct 31 21:20:41.836069 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.835688 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" event={"ID":"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc","Type":"ContainerDied","Data":"b6427ec34b228b961890815a999b7293dae4a87551e666827b5245f242cd672d"} Oct 31 21:20:41.838383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.838341 5188 generic.go:356] "Generic (PLEG): container finished" podID="e25b4385-15fc-4fea-8099-977952f32171" containerID="a651b24e452a7262481b96a25201b7f297c45f15ce4fc56dacd10effadefbd77" exitCode=0 Oct 31 21:20:41.838709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.838424 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" event={"ID":"e25b4385-15fc-4fea-8099-977952f32171","Type":"ContainerDied","Data":"a651b24e452a7262481b96a25201b7f297c45f15ce4fc56dacd10effadefbd77"} Oct 31 21:20:41.841027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.840867 5188 generic.go:356] "Generic (PLEG): container finished" podID="71cf5160-6ca6-45c7-9013-010caa843b72" containerID="1852b34dd336928b424d482139012a17cf7dfaa4e0ba3ffd7ad090ace518f3b6" exitCode=0 Oct 31 21:20:41.841027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.840949 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" event={"ID":"71cf5160-6ca6-45c7-9013-010caa843b72","Type":"ContainerDied","Data":"1852b34dd336928b424d482139012a17cf7dfaa4e0ba3ffd7ad090ace518f3b6"} Oct 31 21:20:41.843553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.843529 5188 generic.go:356] "Generic (PLEG): container finished" podID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerID="264a9d8ef83b6d03ac5e86c1e4f95895369fbafc327ad1b464cdc2dc57567714" exitCode=0 Oct 31 21:20:41.843642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.843614 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" event={"ID":"7f50b1dc-158e-47de-99d3-557e9b2ee04c","Type":"ContainerDied","Data":"264a9d8ef83b6d03ac5e86c1e4f95895369fbafc327ad1b464cdc2dc57567714"} Oct 31 21:20:41.845518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.845495 5188 generic.go:356] "Generic (PLEG): container finished" podID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerID="22da46ff58b66a7a772f91948c7736c17e65c05e4fae758b0ddb13d4bdd39084" exitCode=0 Oct 31 21:20:41.845847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.845595 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" event={"ID":"613dec8c-fe05-4ddc-8a74-75efcba90c90","Type":"ContainerDied","Data":"22da46ff58b66a7a772f91948c7736c17e65c05e4fae758b0ddb13d4bdd39084"} Oct 31 21:20:41.847449 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.847422 5188 generic.go:356] "Generic (PLEG): container finished" podID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerID="c6cc0d0e0860925185dd89e99eda6d0edf403db4f01756180ff74d6acdd9babb" exitCode=0 Oct 31 21:20:41.847558 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:41.847458 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" event={"ID":"ce3b462a-0202-47e2-a2f8-8d061f07ab8c","Type":"ContainerDied","Data":"c6cc0d0e0860925185dd89e99eda6d0edf403db4f01756180ff74d6acdd9babb"} Oct 31 21:20:42.855590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:42.855541 5188 generic.go:356] "Generic (PLEG): container finished" podID="71cf5160-6ca6-45c7-9013-010caa843b72" containerID="d94cdce462a6f37987bc2a4fcd6715e69aca532f43938a100e2d703217a2197e" exitCode=0 Oct 31 21:20:42.856008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:42.855619 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" event={"ID":"71cf5160-6ca6-45c7-9013-010caa843b72","Type":"ContainerDied","Data":"d94cdce462a6f37987bc2a4fcd6715e69aca532f43938a100e2d703217a2197e"} Oct 31 21:20:42.869257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:42.868925 5188 generic.go:356] "Generic (PLEG): container finished" podID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerID="16f366e84442a01230f63828f01bf2e1010a10beff6ffb785be99bbfe213485e" exitCode=0 Oct 31 21:20:42.869257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:42.868976 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" event={"ID":"7f50b1dc-158e-47de-99d3-557e9b2ee04c","Type":"ContainerDied","Data":"16f366e84442a01230f63828f01bf2e1010a10beff6ffb785be99bbfe213485e"} Oct 31 21:20:42.874728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:42.873515 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" event={"ID":"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc","Type":"ContainerStarted","Data":"b543402e50343e27262c4e2c945337c2fe8cc1a320dd6da4bbd140e430d7e979"} Oct 31 21:20:43.884704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:43.883722 5188 generic.go:356] "Generic (PLEG): container finished" podID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerID="5a0ee171b0a6c009ff8f5ad31723a26d00269c75fed812c60362ec878b8c2121" exitCode=0 Oct 31 21:20:43.884704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:43.883820 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" event={"ID":"f7b2a240-2b0f-49b8-b529-c801c62c1160","Type":"ContainerDied","Data":"5a0ee171b0a6c009ff8f5ad31723a26d00269c75fed812c60362ec878b8c2121"} Oct 31 21:20:43.889894 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:43.889842 5188 generic.go:356] "Generic (PLEG): container finished" podID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerID="b543402e50343e27262c4e2c945337c2fe8cc1a320dd6da4bbd140e430d7e979" exitCode=0 Oct 31 21:20:43.890045 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:43.890025 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" event={"ID":"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc","Type":"ContainerDied","Data":"b543402e50343e27262c4e2c945337c2fe8cc1a320dd6da4bbd140e430d7e979"} Oct 31 21:20:43.892996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:43.892960 5188 generic.go:356] "Generic (PLEG): container finished" podID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerID="136cb91f1d5da0f0bb7be30862cccb6e06992ab0b29672feae6431c35e136969" exitCode=0 Oct 31 21:20:43.893101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:43.893001 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" event={"ID":"ce3b462a-0202-47e2-a2f8-8d061f07ab8c","Type":"ContainerDied","Data":"136cb91f1d5da0f0bb7be30862cccb6e06992ab0b29672feae6431c35e136969"} Oct 31 21:20:47.932172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:47.932073 5188 generic.go:356] "Generic (PLEG): container finished" podID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerID="170fd2de3ee0406c90aa38ba0201813237d3a5dc845379bd53b5eb38e0930d64" exitCode=0 Oct 31 21:20:47.932172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:47.932149 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" event={"ID":"613dec8c-fe05-4ddc-8a74-75efcba90c90","Type":"ContainerDied","Data":"170fd2de3ee0406c90aa38ba0201813237d3a5dc845379bd53b5eb38e0930d64"} Oct 31 21:20:53.535400 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:53.535364 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:20:53.537631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:53.537609 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:20:56.003979 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:56.003882 5188 generic.go:356] "Generic (PLEG): container finished" podID="e25b4385-15fc-4fea-8099-977952f32171" containerID="eeb9dcb7b98a6f787f40198ac73d433432c466d81c4965035c6201d8a404229d" exitCode=0 Oct 31 21:20:56.004484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:56.003977 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" event={"ID":"e25b4385-15fc-4fea-8099-977952f32171","Type":"ContainerDied","Data":"eeb9dcb7b98a6f787f40198ac73d433432c466d81c4965035c6201d8a404229d"} Oct 31 21:20:57.217947 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.217896 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:57.344202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.344155 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e25b4385-15fc-4fea-8099-977952f32171-kube-api-access-wdstl\" (UniqueName: \"kubernetes.io/projected/e25b4385-15fc-4fea-8099-977952f32171-kube-api-access-wdstl\") pod \"e25b4385-15fc-4fea-8099-977952f32171\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " Oct 31 21:20:57.344446 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.344220 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-bundle\") pod \"e25b4385-15fc-4fea-8099-977952f32171\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " Oct 31 21:20:57.344446 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.344348 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-util\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-util\") pod \"e25b4385-15fc-4fea-8099-977952f32171\" (UID: \"e25b4385-15fc-4fea-8099-977952f32171\") " Oct 31 21:20:57.344873 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.344841 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-bundle" pod "e25b4385-15fc-4fea-8099-977952f32171" (UID: "e25b4385-15fc-4fea-8099-977952f32171"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:20:57.347176 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.347125 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e25b4385-15fc-4fea-8099-977952f32171-kube-api-access-wdstl" pod "e25b4385-15fc-4fea-8099-977952f32171" (UID: "e25b4385-15fc-4fea-8099-977952f32171"). InnerVolumeSpecName "kube-api-access-wdstl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:20:57.350606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.350567 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-util" pod "e25b4385-15fc-4fea-8099-977952f32171" (UID: "e25b4385-15fc-4fea-8099-977952f32171"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:20:57.446148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.445993 5188 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-util\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:20:57.446148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.446054 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdstl\" (UniqueName: \"kubernetes.io/projected/e25b4385-15fc-4fea-8099-977952f32171-kube-api-access-wdstl\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:20:57.446148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:57.446070 5188 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e25b4385-15fc-4fea-8099-977952f32171-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:20:58.021488 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:58.021461 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" event={"ID":"e25b4385-15fc-4fea-8099-977952f32171","Type":"ContainerDied","Data":"72240cfb5eb92f8b3f5297e280770e965a7d9c44aefee4ce5ddb1f01d08a7ae1"} Oct 31 21:20:58.021575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:58.021511 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72240cfb5eb92f8b3f5297e280770e965a7d9c44aefee4ce5ddb1f01d08a7ae1" Oct 31 21:20:58.021609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:58.021576 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clq8w5" Oct 31 21:20:58.025288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:58.025261 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" event={"ID":"71cf5160-6ca6-45c7-9013-010caa843b72","Type":"ContainerStarted","Data":"4e7c26079c8b3a18bf676b52194e463737776871ef564e7e3cabf3d8b6ce7e66"} Oct 31 21:20:58.028110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:58.028070 5188 generic.go:356] "Generic (PLEG): container finished" podID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerID="5fd29597008891dbd723922ed7f71efd82ff75e844e9d709bcf74e6d2dc05bff" exitCode=0 Oct 31 21:20:58.028231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:58.028194 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" event={"ID":"7f50b1dc-158e-47de-99d3-557e9b2ee04c","Type":"ContainerDied","Data":"5fd29597008891dbd723922ed7f71efd82ff75e844e9d709bcf74e6d2dc05bff"} Oct 31 21:20:58.031825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:58.031793 5188 generic.go:356] "Generic (PLEG): container finished" podID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerID="13963ee326e8d9c0113a9a58dcc45ee4644dbffc0798fa20b6c19cf57fb4b1cf" exitCode=0 Oct 31 21:20:58.031948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:58.031883 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" event={"ID":"613dec8c-fe05-4ddc-8a74-75efcba90c90","Type":"ContainerDied","Data":"13963ee326e8d9c0113a9a58dcc45ee4644dbffc0798fa20b6c19cf57fb4b1cf"} Oct 31 21:20:58.123943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:58.123759 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" podStartSLOduration=2.448481204 podStartE2EDuration="26.123741218s" totalImagesPullingTime="23.675260014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:20:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:20:58.122714184 +0000 UTC m=+304.815671579" watchObservedRunningTime="2026-10-31 21:20:58.123741218 +0000 UTC m=+304.816698612" Oct 31 21:20:59.038138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.038097 5188 generic.go:356] "Generic (PLEG): container finished" podID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerID="0821bc48e7b29e5ea09a5eef76af2b4121d617ccfb07f1061b31943ccf1b423f" exitCode=0 Oct 31 21:20:59.038574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.038177 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" event={"ID":"ce3b462a-0202-47e2-a2f8-8d061f07ab8c","Type":"ContainerDied","Data":"0821bc48e7b29e5ea09a5eef76af2b4121d617ccfb07f1061b31943ccf1b423f"} Oct 31 21:20:59.040207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.040175 5188 generic.go:356] "Generic (PLEG): container finished" podID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerID="8836d796fe7a3188c3089793dd24247d68fc5bc9e2b04dadc6f038ffc93a53fe" exitCode=0 Oct 31 21:20:59.040362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.040205 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" event={"ID":"f7b2a240-2b0f-49b8-b529-c801c62c1160","Type":"ContainerDied","Data":"8836d796fe7a3188c3089793dd24247d68fc5bc9e2b04dadc6f038ffc93a53fe"} Oct 31 21:20:59.042231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.042208 5188 generic.go:356] "Generic (PLEG): container finished" podID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerID="d6b37a5af8bd23784f5900d095c421a540bdfdee1aec018b6b98efe0a1c43e1c" exitCode=0 Oct 31 21:20:59.042337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.042285 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" event={"ID":"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc","Type":"ContainerDied","Data":"d6b37a5af8bd23784f5900d095c421a540bdfdee1aec018b6b98efe0a1c43e1c"} Oct 31 21:20:59.044240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.044217 5188 generic.go:356] "Generic (PLEG): container finished" podID="71cf5160-6ca6-45c7-9013-010caa843b72" containerID="4e7c26079c8b3a18bf676b52194e463737776871ef564e7e3cabf3d8b6ce7e66" exitCode=0 Oct 31 21:20:59.044352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.044283 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" event={"ID":"71cf5160-6ca6-45c7-9013-010caa843b72","Type":"ContainerDied","Data":"4e7c26079c8b3a18bf676b52194e463737776871ef564e7e3cabf3d8b6ce7e66"} Oct 31 21:20:59.187875 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.187849 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:20:59.216461 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.216399 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:20:59.261663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.261631 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-bundle\") pod \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " Oct 31 21:20:59.261663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.261671 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-util\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-util\") pod \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " Oct 31 21:20:59.261963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.261693 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-bundle\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-bundle\") pod \"613dec8c-fe05-4ddc-8a74-75efcba90c90\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " Oct 31 21:20:59.261963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.261733 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f50b1dc-158e-47de-99d3-557e9b2ee04c-kube-api-access-k7x6x\" (UniqueName: \"kubernetes.io/projected/7f50b1dc-158e-47de-99d3-557e9b2ee04c-kube-api-access-k7x6x\") pod \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\" (UID: \"7f50b1dc-158e-47de-99d3-557e9b2ee04c\") " Oct 31 21:20:59.261963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.261771 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613dec8c-fe05-4ddc-8a74-75efcba90c90-kube-api-access-85n9s\" (UniqueName: \"kubernetes.io/projected/613dec8c-fe05-4ddc-8a74-75efcba90c90-kube-api-access-85n9s\") pod \"613dec8c-fe05-4ddc-8a74-75efcba90c90\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " Oct 31 21:20:59.261963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.261797 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-util\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-util\") pod \"613dec8c-fe05-4ddc-8a74-75efcba90c90\" (UID: \"613dec8c-fe05-4ddc-8a74-75efcba90c90\") " Oct 31 21:20:59.262316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.262270 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-bundle" pod "613dec8c-fe05-4ddc-8a74-75efcba90c90" (UID: "613dec8c-fe05-4ddc-8a74-75efcba90c90"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:20:59.262316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.262497 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-bundle" pod "7f50b1dc-158e-47de-99d3-557e9b2ee04c" (UID: "7f50b1dc-158e-47de-99d3-557e9b2ee04c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:20:59.264646 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.264618 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613dec8c-fe05-4ddc-8a74-75efcba90c90-kube-api-access-85n9s" pod "613dec8c-fe05-4ddc-8a74-75efcba90c90" (UID: "613dec8c-fe05-4ddc-8a74-75efcba90c90"). InnerVolumeSpecName "kube-api-access-85n9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:20:59.264743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.264667 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f50b1dc-158e-47de-99d3-557e9b2ee04c-kube-api-access-k7x6x" pod "7f50b1dc-158e-47de-99d3-557e9b2ee04c" (UID: "7f50b1dc-158e-47de-99d3-557e9b2ee04c"). InnerVolumeSpecName "kube-api-access-k7x6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:20:59.267948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.267924 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-util" pod "7f50b1dc-158e-47de-99d3-557e9b2ee04c" (UID: "7f50b1dc-158e-47de-99d3-557e9b2ee04c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:20:59.270410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.270389 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-util" pod "613dec8c-fe05-4ddc-8a74-75efcba90c90" (UID: "613dec8c-fe05-4ddc-8a74-75efcba90c90"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:20:59.363263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.363225 5188 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:20:59.363263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.363257 5188 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f50b1dc-158e-47de-99d3-557e9b2ee04c-util\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:20:59.363263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.363266 5188 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:20:59.363541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.363274 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7x6x\" (UniqueName: \"kubernetes.io/projected/7f50b1dc-158e-47de-99d3-557e9b2ee04c-kube-api-access-k7x6x\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:20:59.363541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.363288 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85n9s\" (UniqueName: \"kubernetes.io/projected/613dec8c-fe05-4ddc-8a74-75efcba90c90-kube-api-access-85n9s\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:20:59.363541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:20:59.363297 5188 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/613dec8c-fe05-4ddc-8a74-75efcba90c90-util\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.050984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.050945 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" event={"ID":"7f50b1dc-158e-47de-99d3-557e9b2ee04c","Type":"ContainerDied","Data":"072dc5b74d61562be3a0c4aaec715e9b8ed575b5e1e529616741d5beff0df539"} Oct 31 21:21:00.051445 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.050999 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="072dc5b74d61562be3a0c4aaec715e9b8ed575b5e1e529616741d5beff0df539" Oct 31 21:21:00.051445 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.050963 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpnqx" Oct 31 21:21:00.053156 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.053131 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" Oct 31 21:21:00.053267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.053203 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6445bcjh" event={"ID":"613dec8c-fe05-4ddc-8a74-75efcba90c90","Type":"ContainerDied","Data":"2aa472403456e4c660a61223795bed9cee73ed22ac39f7c4421cbb3f1a21cd1a"} Oct 31 21:21:00.053267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.053255 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aa472403456e4c660a61223795bed9cee73ed22ac39f7c4421cbb3f1a21cd1a" Oct 31 21:21:00.230942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.230833 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:21:00.263695 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.263665 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:21:00.267530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.267505 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:21:00.271291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.271264 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:21:00.371821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.371789 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-kube-api-access-qtjpr\" (UniqueName: \"kubernetes.io/projected/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-kube-api-access-qtjpr\") pod \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " Oct 31 21:21:00.371998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.371827 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-kube-api-access-rkk94\" (UniqueName: \"kubernetes.io/projected/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-kube-api-access-rkk94\") pod \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " Oct 31 21:21:00.371998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.371872 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-util\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-util\") pod \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " Oct 31 21:21:00.371998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.371909 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-util\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-util\") pod \"f7b2a240-2b0f-49b8-b529-c801c62c1160\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " Oct 31 21:21:00.371998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.371933 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-bundle\") pod \"f7b2a240-2b0f-49b8-b529-c801c62c1160\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " Oct 31 21:21:00.371998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.371974 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b2a240-2b0f-49b8-b529-c801c62c1160-kube-api-access-2vc9g\" (UniqueName: \"kubernetes.io/projected/f7b2a240-2b0f-49b8-b529-c801c62c1160-kube-api-access-2vc9g\") pod \"f7b2a240-2b0f-49b8-b529-c801c62c1160\" (UID: \"f7b2a240-2b0f-49b8-b529-c801c62c1160\") " Oct 31 21:21:00.372271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.372017 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cf5160-6ca6-45c7-9013-010caa843b72-kube-api-access-2zr9q\" (UniqueName: \"kubernetes.io/projected/71cf5160-6ca6-45c7-9013-010caa843b72-kube-api-access-2zr9q\") pod \"71cf5160-6ca6-45c7-9013-010caa843b72\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " Oct 31 21:21:00.372271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.372057 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-bundle\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-bundle\") pod \"71cf5160-6ca6-45c7-9013-010caa843b72\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " Oct 31 21:21:00.372271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.372082 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-bundle\") pod \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\" (UID: \"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc\") " Oct 31 21:21:00.372271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.372114 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-bundle\") pod \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " Oct 31 21:21:00.372271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.372171 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-util\") pod \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\" (UID: \"ce3b462a-0202-47e2-a2f8-8d061f07ab8c\") " Oct 31 21:21:00.372271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.372225 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-util\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-util\") pod \"71cf5160-6ca6-45c7-9013-010caa843b72\" (UID: \"71cf5160-6ca6-45c7-9013-010caa843b72\") " Oct 31 21:21:00.375584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.372778 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-bundle" pod "f7b2a240-2b0f-49b8-b529-c801c62c1160" (UID: "f7b2a240-2b0f-49b8-b529-c801c62c1160"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:21:00.375584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.373539 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-bundle" pod "71cf5160-6ca6-45c7-9013-010caa843b72" (UID: "71cf5160-6ca6-45c7-9013-010caa843b72"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:21:00.375584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.374228 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-bundle" pod "ce3b462a-0202-47e2-a2f8-8d061f07ab8c" (UID: "ce3b462a-0202-47e2-a2f8-8d061f07ab8c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:21:00.375584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.374663 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-kube-api-access-qtjpr" pod "bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" (UID: "bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc"). InnerVolumeSpecName "kube-api-access-qtjpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:21:00.375584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.374920 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-bundle" pod "bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" (UID: "bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:21:00.375584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.375030 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-kube-api-access-rkk94" pod "ce3b462a-0202-47e2-a2f8-8d061f07ab8c" (UID: "ce3b462a-0202-47e2-a2f8-8d061f07ab8c"). InnerVolumeSpecName "kube-api-access-rkk94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:21:00.377063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.377034 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cf5160-6ca6-45c7-9013-010caa843b72-kube-api-access-2zr9q" pod "71cf5160-6ca6-45c7-9013-010caa843b72" (UID: "71cf5160-6ca6-45c7-9013-010caa843b72"). InnerVolumeSpecName "kube-api-access-2zr9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:21:00.378275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.378236 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b2a240-2b0f-49b8-b529-c801c62c1160-kube-api-access-2vc9g" pod "f7b2a240-2b0f-49b8-b529-c801c62c1160" (UID: "f7b2a240-2b0f-49b8-b529-c801c62c1160"). InnerVolumeSpecName "kube-api-access-2vc9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:21:00.378498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.378470 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-util" pod "bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" (UID: "bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:21:00.379043 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.379018 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-util" pod "f7b2a240-2b0f-49b8-b529-c801c62c1160" (UID: "f7b2a240-2b0f-49b8-b529-c801c62c1160"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:21:00.380302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.380277 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-util" pod "71cf5160-6ca6-45c7-9013-010caa843b72" (UID: "71cf5160-6ca6-45c7-9013-010caa843b72"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:21:00.380920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.380897 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-util" pod "ce3b462a-0202-47e2-a2f8-8d061f07ab8c" (UID: "ce3b462a-0202-47e2-a2f8-8d061f07ab8c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:21:00.473181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473137 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtjpr\" (UniqueName: \"kubernetes.io/projected/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-kube-api-access-qtjpr\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473174 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkk94\" (UniqueName: \"kubernetes.io/projected/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-kube-api-access-rkk94\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473185 5188 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-util\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473195 5188 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-util\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473203 5188 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7b2a240-2b0f-49b8-b529-c801c62c1160-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473211 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vc9g\" (UniqueName: \"kubernetes.io/projected/f7b2a240-2b0f-49b8-b529-c801c62c1160-kube-api-access-2vc9g\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473220 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zr9q\" (UniqueName: \"kubernetes.io/projected/71cf5160-6ca6-45c7-9013-010caa843b72-kube-api-access-2zr9q\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473228 5188 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473236 5188 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473244 5188 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-bundle\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473251 5188 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ce3b462a-0202-47e2-a2f8-8d061f07ab8c-util\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:00.473455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:00.473258 5188 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71cf5160-6ca6-45c7-9013-010caa843b72-util\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:01.058828 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.058793 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" event={"ID":"f7b2a240-2b0f-49b8-b529-c801c62c1160","Type":"ContainerDied","Data":"a854288c77efec377effb5a656f9220df2dd5e43748a8d637ee9122850e15f4f"} Oct 31 21:21:01.059226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.058851 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a854288c77efec377effb5a656f9220df2dd5e43748a8d637ee9122850e15f4f" Oct 31 21:21:01.059226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.058802 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpg69v" Oct 31 21:21:01.060847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.060820 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" Oct 31 21:21:01.060982 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.060862 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79nvdl" event={"ID":"bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc","Type":"ContainerDied","Data":"a48bedbe0004f163cb7ea0f3edd74ae9dcf078aab0529b710c8c5e4ccf7316d5"} Oct 31 21:21:01.060982 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.060903 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a48bedbe0004f163cb7ea0f3edd74ae9dcf078aab0529b710c8c5e4ccf7316d5" Oct 31 21:21:01.063012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.062988 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" Oct 31 21:21:01.063012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.063001 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pzfvb" event={"ID":"71cf5160-6ca6-45c7-9013-010caa843b72","Type":"ContainerDied","Data":"327ef882c9358caf987634d0b04ceae194492b7be68ca52465bb58253da2ee27"} Oct 31 21:21:01.063185 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.063047 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="327ef882c9358caf987634d0b04ceae194492b7be68ca52465bb58253da2ee27" Oct 31 21:21:01.065089 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.065068 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" event={"ID":"ce3b462a-0202-47e2-a2f8-8d061f07ab8c","Type":"ContainerDied","Data":"da39aa4f7d9c45c19fd0431256568b98fd92db2cd0ba97b346d2153288d48777"} Oct 31 21:21:01.065178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.065107 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da39aa4f7d9c45c19fd0431256568b98fd92db2cd0ba97b346d2153288d48777" Oct 31 21:21:01.065178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:01.065114 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847abz99d" Oct 31 21:21:02.535848 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:21:02.535807 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:21:05.013178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013097 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp"] Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013459 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013472 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013484 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013489 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013498 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerName="extract" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013504 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerName="extract" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013516 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerName="util" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013521 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerName="util" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013529 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerName="util" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013533 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerName="util" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013540 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013544 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013553 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013557 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013565 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e25b4385-15fc-4fea-8099-977952f32171" containerName="util" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013570 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25b4385-15fc-4fea-8099-977952f32171" containerName="util" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013577 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e25b4385-15fc-4fea-8099-977952f32171" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013581 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25b4385-15fc-4fea-8099-977952f32171" containerName="pull" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013590 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerName="util" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013595 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerName="util" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013601 5188 cpu_manager.go:414] "containerMap: removing container" podUID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerName="extract" Oct 31 21:21:05.013596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013606 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013613 5188 cpu_manager.go:414] "containerMap: removing container" podUID="71cf5160-6ca6-45c7-9013-010caa843b72" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013617 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cf5160-6ca6-45c7-9013-010caa843b72" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013624 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013628 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013636 5188 cpu_manager.go:414] "containerMap: removing container" podUID="71cf5160-6ca6-45c7-9013-010caa843b72" containerName="pull" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013640 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cf5160-6ca6-45c7-9013-010caa843b72" containerName="pull" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013650 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerName="util" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013654 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerName="util" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013659 5188 cpu_manager.go:414] "containerMap: removing container" podUID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerName="pull" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013663 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerName="pull" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013671 5188 cpu_manager.go:414] "containerMap: removing container" podUID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerName="util" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013675 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerName="util" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013681 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e25b4385-15fc-4fea-8099-977952f32171" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013685 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e25b4385-15fc-4fea-8099-977952f32171" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013691 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013695 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013703 5188 cpu_manager.go:414] "containerMap: removing container" podUID="71cf5160-6ca6-45c7-9013-010caa843b72" containerName="util" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013707 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cf5160-6ca6-45c7-9013-010caa843b72" containerName="util" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013713 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013717 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013765 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e25b4385-15fc-4fea-8099-977952f32171" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013772 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3b462a-0202-47e2-a2f8-8d061f07ab8c" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013777 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f50b1dc-158e-47de-99d3-557e9b2ee04c" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013784 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b2a240-2b0f-49b8-b529-c801c62c1160" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013790 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="613dec8c-fe05-4ddc-8a74-75efcba90c90" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013796 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cf5160-6ca6-45c7-9013-010caa843b72" containerName="extract" Oct 31 21:21:05.014240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.013800 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8bf7ad-6cc8-4838-952a-a6d13b57f5fc" containerName="extract" Oct 31 21:21:05.036056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.036018 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" Oct 31 21:21:05.037985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.037954 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp"] Oct 31 21:21:05.048862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.048837 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-9vf8g\"" Oct 31 21:21:05.058665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.058636 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:21:05.058817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.058684 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 21:21:05.114138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.114099 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmdg\" (UniqueName: \"kubernetes.io/projected/dc66dac4-5017-4f11-9f58-b0f20b6d5a9d-kube-api-access-jxmdg\") pod \"jobset-operator-674cd59dc5-d5mxp\" (UID: \"dc66dac4-5017-4f11-9f58-b0f20b6d5a9d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" Oct 31 21:21:05.114355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.114151 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc66dac4-5017-4f11-9f58-b0f20b6d5a9d-tmp\") pod \"jobset-operator-674cd59dc5-d5mxp\" (UID: \"dc66dac4-5017-4f11-9f58-b0f20b6d5a9d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" Oct 31 21:21:05.215184 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.215140 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmdg\" (UniqueName: \"kubernetes.io/projected/dc66dac4-5017-4f11-9f58-b0f20b6d5a9d-kube-api-access-jxmdg\") pod \"jobset-operator-674cd59dc5-d5mxp\" (UID: \"dc66dac4-5017-4f11-9f58-b0f20b6d5a9d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" Oct 31 21:21:05.215370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.215197 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc66dac4-5017-4f11-9f58-b0f20b6d5a9d-tmp\") pod \"jobset-operator-674cd59dc5-d5mxp\" (UID: \"dc66dac4-5017-4f11-9f58-b0f20b6d5a9d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" Oct 31 21:21:05.215700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.215678 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc66dac4-5017-4f11-9f58-b0f20b6d5a9d-tmp\") pod \"jobset-operator-674cd59dc5-d5mxp\" (UID: \"dc66dac4-5017-4f11-9f58-b0f20b6d5a9d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" Oct 31 21:21:05.243219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.243181 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmdg\" (UniqueName: \"kubernetes.io/projected/dc66dac4-5017-4f11-9f58-b0f20b6d5a9d-kube-api-access-jxmdg\") pod \"jobset-operator-674cd59dc5-d5mxp\" (UID: \"dc66dac4-5017-4f11-9f58-b0f20b6d5a9d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" Oct 31 21:21:05.382093 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.382059 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" Oct 31 21:21:05.699880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.699781 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp"] Oct 31 21:21:05.702660 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:21:05.702624 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc66dac4_5017_4f11_9f58_b0f20b6d5a9d.slice/crio-ad56bf0200442bd64b7d0166582e338a26e8fe900d739491452decf2cb31797f WatchSource:0}: Error finding container ad56bf0200442bd64b7d0166582e338a26e8fe900d739491452decf2cb31797f: Status 404 returned error can't find the container with id ad56bf0200442bd64b7d0166582e338a26e8fe900d739491452decf2cb31797f Oct 31 21:21:05.704272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:05.704251 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:21:06.088897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:06.088813 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" event={"ID":"dc66dac4-5017-4f11-9f58-b0f20b6d5a9d","Type":"ContainerStarted","Data":"ad56bf0200442bd64b7d0166582e338a26e8fe900d739491452decf2cb31797f"} Oct 31 21:21:08.099857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:08.099819 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" event={"ID":"dc66dac4-5017-4f11-9f58-b0f20b6d5a9d","Type":"ContainerStarted","Data":"151cb4ee7b0a2b2e39dac186141c4771ac52d4d0f19503170ca07bc758d9a5ba"} Oct 31 21:21:08.142702 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:08.142643 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-d5mxp" podStartSLOduration=2.489694375 podStartE2EDuration="4.142629537s" totalImagesPullingTime="1.652935162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:08.140691876 +0000 UTC m=+314.833649269" watchObservedRunningTime="2026-10-31 21:21:08.142629537 +0000 UTC m=+314.835586930" Oct 31 21:21:12.041638 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.041596 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf"] Oct 31 21:21:12.045276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.045239 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.058909 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.058861 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 21:21:12.059265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.059240 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 21:21:12.059265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.059241 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 21:21:12.059457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.059420 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lmxhd\"" Oct 31 21:21:12.068774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.068745 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 21:21:12.108370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.108314 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf"] Oct 31 21:21:12.159521 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.159477 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 21:21:12.176914 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.176869 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-tmp\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.177096 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.176949 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-metrics-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.177096 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.177010 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgvsg\" (UniqueName: \"kubernetes.io/projected/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-kube-api-access-lgvsg\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.177096 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.177052 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-manifests\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.177208 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.177140 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.177208 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.177195 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.277644 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.277599 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.277644 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.277643 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-tmp\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.277916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.277697 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-metrics-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.277916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.277724 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvsg\" (UniqueName: \"kubernetes.io/projected/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-kube-api-access-lgvsg\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.277916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.277771 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-manifests\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.277916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.277812 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.278278 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.278245 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-manifests\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.280297 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.280267 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-tmp\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.280602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.280578 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.280665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.280581 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-webhook-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.280665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.280605 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-metrics-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.301674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.301592 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvsg\" (UniqueName: \"kubernetes.io/projected/1e5724bb-bfd2-4659-ba01-de380bf9d7d2-kube-api-access-lgvsg\") pod \"opendatahub-operator-controller-manager-899c649cb-7cjsf\" (UID: \"1e5724bb-bfd2-4659-ba01-de380bf9d7d2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.361991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.361955 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:12.730208 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:12.730181 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf"] Oct 31 21:21:12.732052 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:21:12.732024 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5724bb_bfd2_4659_ba01_de380bf9d7d2.slice/crio-f258bd6eb35680ee54e369614b52dd7041349c2df84e60983c7ec138524753e2 WatchSource:0}: Error finding container f258bd6eb35680ee54e369614b52dd7041349c2df84e60983c7ec138524753e2: Status 404 returned error can't find the container with id f258bd6eb35680ee54e369614b52dd7041349c2df84e60983c7ec138524753e2 Oct 31 21:21:13.124166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:13.124123 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" event={"ID":"1e5724bb-bfd2-4659-ba01-de380bf9d7d2","Type":"ContainerStarted","Data":"f258bd6eb35680ee54e369614b52dd7041349c2df84e60983c7ec138524753e2"} Oct 31 21:21:16.142133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:16.142095 5188 generic.go:356] "Generic (PLEG): container finished" podID="1e5724bb-bfd2-4659-ba01-de380bf9d7d2" containerID="5e744fd3b11a7fbeb5b3ee520a6cfb7c60c6b75f4281fd6a4d7fca17459a1aa0" exitCode=0 Oct 31 21:21:16.142617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:16.142176 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" event={"ID":"1e5724bb-bfd2-4659-ba01-de380bf9d7d2","Type":"ContainerDied","Data":"5e744fd3b11a7fbeb5b3ee520a6cfb7c60c6b75f4281fd6a4d7fca17459a1aa0"} Oct 31 21:21:17.148498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:17.148462 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" event={"ID":"1e5724bb-bfd2-4659-ba01-de380bf9d7d2","Type":"ContainerStarted","Data":"82c7f4b915ed540cefd2208e55d347ace94edc59181350e5560e87db38da4e2e"} Oct 31 21:21:17.148962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:17.148574 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:17.230523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:17.230461 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" podStartSLOduration=2.756826463 podStartE2EDuration="6.230444442s" totalImagesPullingTime="3.041732688s" totalInitContainerRuntime="431.885291ms" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:17.220047505 +0000 UTC m=+323.913004910" watchObservedRunningTime="2026-10-31 21:21:17.230444442 +0000 UTC m=+323.923401834" Oct 31 21:21:18.669907 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.669866 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw"] Oct 31 21:21:18.673733 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.673708 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" Oct 31 21:21:18.692392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.692353 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 21:21:18.701387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.701350 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 21:21:18.701387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.701373 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zfqhb\"" Oct 31 21:21:18.703334 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.703291 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw"] Oct 31 21:21:18.834482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.834444 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8tk4\" (UniqueName: \"kubernetes.io/projected/71e739c4-f697-4b84-9b3f-1ee74b73a01b-kube-api-access-c8tk4\") pod \"authorino-operator-7b7bb4bc77-l7tsw\" (UID: \"71e739c4-f697-4b84-9b3f-1ee74b73a01b\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" Oct 31 21:21:18.935816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.935718 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8tk4\" (UniqueName: \"kubernetes.io/projected/71e739c4-f697-4b84-9b3f-1ee74b73a01b-kube-api-access-c8tk4\") pod \"authorino-operator-7b7bb4bc77-l7tsw\" (UID: \"71e739c4-f697-4b84-9b3f-1ee74b73a01b\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" Oct 31 21:21:18.958003 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.957961 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8tk4\" (UniqueName: \"kubernetes.io/projected/71e739c4-f697-4b84-9b3f-1ee74b73a01b-kube-api-access-c8tk4\") pod \"authorino-operator-7b7bb4bc77-l7tsw\" (UID: \"71e739c4-f697-4b84-9b3f-1ee74b73a01b\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" Oct 31 21:21:18.986794 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:18.986754 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" Oct 31 21:21:19.293266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:19.293234 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw"] Oct 31 21:21:19.294848 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:21:19.294821 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e739c4_f697_4b84_9b3f_1ee74b73a01b.slice/crio-27a8634ecb1a354b8a1cc3f1f3fef4a7eb38a10333b1d64003a369764110df29 WatchSource:0}: Error finding container 27a8634ecb1a354b8a1cc3f1f3fef4a7eb38a10333b1d64003a369764110df29: Status 404 returned error can't find the container with id 27a8634ecb1a354b8a1cc3f1f3fef4a7eb38a10333b1d64003a369764110df29 Oct 31 21:21:20.165067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:20.164997 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" event={"ID":"71e739c4-f697-4b84-9b3f-1ee74b73a01b","Type":"ContainerStarted","Data":"27a8634ecb1a354b8a1cc3f1f3fef4a7eb38a10333b1d64003a369764110df29"} Oct 31 21:21:22.177621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:22.177575 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" event={"ID":"71e739c4-f697-4b84-9b3f-1ee74b73a01b","Type":"ContainerStarted","Data":"3ccca841ac6d2cbd2567b1d51dbe2c6095a01edaa228b319be20ced4d7f768f9"} Oct 31 21:21:22.178051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:22.177639 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" Oct 31 21:21:22.218659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:22.218590 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" podStartSLOduration=1.871022919 podStartE2EDuration="4.218571455s" totalImagesPullingTime="2.347548536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:22.217282594 +0000 UTC m=+328.910239988" watchObservedRunningTime="2026-10-31 21:21:22.218571455 +0000 UTC m=+328.911528848" Oct 31 21:21:26.473886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.473849 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb"] Oct 31 21:21:26.478721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.478700 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:26.491896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.491869 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kjkrh\"" Oct 31 21:21:26.503627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.503594 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb"] Oct 31 21:21:26.603767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.603724 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8d62927-0be1-4ab5-8961-f28340b349c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cxhcb\" (UID: \"e8d62927-0be1-4ab5-8961-f28340b349c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:26.603963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.603782 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmr2\" (UniqueName: \"kubernetes.io/projected/e8d62927-0be1-4ab5-8961-f28340b349c3-kube-api-access-kbmr2\") pod \"kuadrant-operator-controller-manager-678d87766f-cxhcb\" (UID: \"e8d62927-0be1-4ab5-8961-f28340b349c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:26.704536 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.704486 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmr2\" (UniqueName: \"kubernetes.io/projected/e8d62927-0be1-4ab5-8961-f28340b349c3-kube-api-access-kbmr2\") pod \"kuadrant-operator-controller-manager-678d87766f-cxhcb\" (UID: \"e8d62927-0be1-4ab5-8961-f28340b349c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:26.704760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.704594 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8d62927-0be1-4ab5-8961-f28340b349c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cxhcb\" (UID: \"e8d62927-0be1-4ab5-8961-f28340b349c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:26.704998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.704972 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8d62927-0be1-4ab5-8961-f28340b349c3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-cxhcb\" (UID: \"e8d62927-0be1-4ab5-8961-f28340b349c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:26.747077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.747009 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmr2\" (UniqueName: \"kubernetes.io/projected/e8d62927-0be1-4ab5-8961-f28340b349c3-kube-api-access-kbmr2\") pod \"kuadrant-operator-controller-manager-678d87766f-cxhcb\" (UID: \"e8d62927-0be1-4ab5-8961-f28340b349c3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:26.792753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:26.792715 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:27.133991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:27.133925 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb"] Oct 31 21:21:27.136257 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:21:27.136225 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d62927_0be1_4ab5_8961_f28340b349c3.slice/crio-6fea79fd40e58c47626705afba9fef1bf1b745cf828bb85e6fe3f04086bf3e70 WatchSource:0}: Error finding container 6fea79fd40e58c47626705afba9fef1bf1b745cf828bb85e6fe3f04086bf3e70: Status 404 returned error can't find the container with id 6fea79fd40e58c47626705afba9fef1bf1b745cf828bb85e6fe3f04086bf3e70 Oct 31 21:21:27.201438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:27.201394 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" event={"ID":"e8d62927-0be1-4ab5-8961-f28340b349c3","Type":"ContainerStarted","Data":"6fea79fd40e58c47626705afba9fef1bf1b745cf828bb85e6fe3f04086bf3e70"} Oct 31 21:21:28.154984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:28.154951 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-7cjsf" Oct 31 21:21:32.537542 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:21:32.537497 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:21:33.184583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:33.184550 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-l7tsw" Oct 31 21:21:33.235619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:33.235585 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" event={"ID":"e8d62927-0be1-4ab5-8961-f28340b349c3","Type":"ContainerStarted","Data":"0339515e8553b7d2e4fa03ed5a91259be5aca20b1323e91abca90d57a802f378"} Oct 31 21:21:33.235824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:33.235774 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:33.277669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:33.277608 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" podStartSLOduration=2.069673042 podStartE2EDuration="7.277592269s" totalImagesPullingTime="5.207919227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:33.276380768 +0000 UTC m=+339.969338160" watchObservedRunningTime="2026-10-31 21:21:33.277592269 +0000 UTC m=+339.970549656" Oct 31 21:21:35.769767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.769726 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf"] Oct 31 21:21:35.773082 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.773060 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:35.792706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.792646 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf"] Oct 31 21:21:35.797684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.797648 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 21:21:35.797684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.797667 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 21:21:35.860799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.860765 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hd6xn\"" Oct 31 21:21:35.873276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.873240 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75c7n\" (UniqueName: \"kubernetes.io/projected/e606927e-6418-4c79-bd40-6ef9cbfec905-kube-api-access-75c7n\") pod \"cert-manager-webhook-5dbbbf75cc-s6zvf\" (UID: \"e606927e-6418-4c79-bd40-6ef9cbfec905\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:35.873276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.873285 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e606927e-6418-4c79-bd40-6ef9cbfec905-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-s6zvf\" (UID: \"e606927e-6418-4c79-bd40-6ef9cbfec905\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:35.974218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.974176 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75c7n\" (UniqueName: \"kubernetes.io/projected/e606927e-6418-4c79-bd40-6ef9cbfec905-kube-api-access-75c7n\") pod \"cert-manager-webhook-5dbbbf75cc-s6zvf\" (UID: \"e606927e-6418-4c79-bd40-6ef9cbfec905\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:35.974218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.974222 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e606927e-6418-4c79-bd40-6ef9cbfec905-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-s6zvf\" (UID: \"e606927e-6418-4c79-bd40-6ef9cbfec905\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:35.995590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.995561 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e606927e-6418-4c79-bd40-6ef9cbfec905-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-s6zvf\" (UID: \"e606927e-6418-4c79-bd40-6ef9cbfec905\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:35.995728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:35.995702 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75c7n\" (UniqueName: \"kubernetes.io/projected/e606927e-6418-4c79-bd40-6ef9cbfec905-kube-api-access-75c7n\") pod \"cert-manager-webhook-5dbbbf75cc-s6zvf\" (UID: \"e606927e-6418-4c79-bd40-6ef9cbfec905\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:36.085681 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:36.085577 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:36.416372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:36.416343 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf"] Oct 31 21:21:36.418347 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:21:36.418302 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode606927e_6418_4c79_bd40_6ef9cbfec905.slice/crio-9b2316f0af529ebd0eefef71b643db93daf6f40cacc284f53e55c19ff09850cb WatchSource:0}: Error finding container 9b2316f0af529ebd0eefef71b643db93daf6f40cacc284f53e55c19ff09850cb: Status 404 returned error can't find the container with id 9b2316f0af529ebd0eefef71b643db93daf6f40cacc284f53e55c19ff09850cb Oct 31 21:21:37.257111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:37.257069 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" event={"ID":"e606927e-6418-4c79-bd40-6ef9cbfec905","Type":"ContainerStarted","Data":"9b2316f0af529ebd0eefef71b643db93daf6f40cacc284f53e55c19ff09850cb"} Oct 31 21:21:39.269041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.268932 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" event={"ID":"e606927e-6418-4c79-bd40-6ef9cbfec905","Type":"ContainerStarted","Data":"46cbf4310e414c163e79dd68356a7209600e2049a07e83fb46b464996a4b5df1"} Oct 31 21:21:39.269041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.269007 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:39.312132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.312078 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" podStartSLOduration=1.560561477 podStartE2EDuration="4.312062885s" totalImagesPullingTime="2.751501408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:39.309817401 +0000 UTC m=+346.002774793" watchObservedRunningTime="2026-10-31 21:21:39.312062885 +0000 UTC m=+346.005020275" Oct 31 21:21:39.335240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.335201 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs"] Oct 31 21:21:39.339026 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.339003 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.355590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.355556 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 21:21:39.355590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.355565 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p69xn\"" Oct 31 21:21:39.355813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.355569 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 21:21:39.362796 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.360206 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs"] Oct 31 21:21:39.402871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.402839 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc9113e0-2ee8-44de-af57-00602a936fe6-tmp\") pod \"kube-auth-proxy-5587c8dd5c-htpqs\" (UID: \"dc9113e0-2ee8-44de-af57-00602a936fe6\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.403054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.402874 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psjkq\" (UniqueName: \"kubernetes.io/projected/dc9113e0-2ee8-44de-af57-00602a936fe6-kube-api-access-psjkq\") pod \"kube-auth-proxy-5587c8dd5c-htpqs\" (UID: \"dc9113e0-2ee8-44de-af57-00602a936fe6\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.403054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.402975 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9113e0-2ee8-44de-af57-00602a936fe6-tls-certs\") pod \"kube-auth-proxy-5587c8dd5c-htpqs\" (UID: \"dc9113e0-2ee8-44de-af57-00602a936fe6\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.504086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.503986 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9113e0-2ee8-44de-af57-00602a936fe6-tls-certs\") pod \"kube-auth-proxy-5587c8dd5c-htpqs\" (UID: \"dc9113e0-2ee8-44de-af57-00602a936fe6\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.504086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.504043 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc9113e0-2ee8-44de-af57-00602a936fe6-tmp\") pod \"kube-auth-proxy-5587c8dd5c-htpqs\" (UID: \"dc9113e0-2ee8-44de-af57-00602a936fe6\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.504313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.504164 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psjkq\" (UniqueName: \"kubernetes.io/projected/dc9113e0-2ee8-44de-af57-00602a936fe6-kube-api-access-psjkq\") pod \"kube-auth-proxy-5587c8dd5c-htpqs\" (UID: \"dc9113e0-2ee8-44de-af57-00602a936fe6\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.506456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.506432 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc9113e0-2ee8-44de-af57-00602a936fe6-tmp\") pod \"kube-auth-proxy-5587c8dd5c-htpqs\" (UID: \"dc9113e0-2ee8-44de-af57-00602a936fe6\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.506721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.506701 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9113e0-2ee8-44de-af57-00602a936fe6-tls-certs\") pod \"kube-auth-proxy-5587c8dd5c-htpqs\" (UID: \"dc9113e0-2ee8-44de-af57-00602a936fe6\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.533666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.533631 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psjkq\" (UniqueName: \"kubernetes.io/projected/dc9113e0-2ee8-44de-af57-00602a936fe6-kube-api-access-psjkq\") pod \"kube-auth-proxy-5587c8dd5c-htpqs\" (UID: \"dc9113e0-2ee8-44de-af57-00602a936fe6\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.651428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.651392 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" Oct 31 21:21:39.963094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:39.963067 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs"] Oct 31 21:21:39.965442 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:21:39.965412 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9113e0_2ee8_44de_af57_00602a936fe6.slice/crio-97f6756b9273297cbf16074b04f187f11062b6f8b3cb78786f58fa0c661333c2 WatchSource:0}: Error finding container 97f6756b9273297cbf16074b04f187f11062b6f8b3cb78786f58fa0c661333c2: Status 404 returned error can't find the container with id 97f6756b9273297cbf16074b04f187f11062b6f8b3cb78786f58fa0c661333c2 Oct 31 21:21:40.274611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:40.274520 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" event={"ID":"dc9113e0-2ee8-44de-af57-00602a936fe6","Type":"ContainerStarted","Data":"97f6756b9273297cbf16074b04f187f11062b6f8b3cb78786f58fa0c661333c2"} Oct 31 21:21:43.143189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.143148 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5nrtq"] Oct 31 21:21:43.146835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.146809 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" Oct 31 21:21:43.164370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.164302 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5nrtq"] Oct 31 21:21:43.187489 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.187434 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-956cc\"" Oct 31 21:21:43.237071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.237036 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kglrd\" (UniqueName: \"kubernetes.io/projected/c5aefada-2f47-4798-9b15-950327db9222-kube-api-access-kglrd\") pod \"authorino-57f7f8659f-5nrtq\" (UID: \"c5aefada-2f47-4798-9b15-950327db9222\") " pod="kuadrant-system/authorino-57f7f8659f-5nrtq" Oct 31 21:21:43.291981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.291935 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" event={"ID":"dc9113e0-2ee8-44de-af57-00602a936fe6","Type":"ContainerStarted","Data":"035d0377e9edf29e58f7ec1b1c557301465b7ce5a2bd6c70580c7f117ce44b8c"} Oct 31 21:21:43.331918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.331850 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-htpqs" podStartSLOduration=1.829356535 podStartE2EDuration="4.331834451s" totalImagesPullingTime="2.502477916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:43.330426479 +0000 UTC m=+350.023383872" watchObservedRunningTime="2026-10-31 21:21:43.331834451 +0000 UTC m=+350.024791844" Oct 31 21:21:43.337924 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.337862 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kglrd\" (UniqueName: \"kubernetes.io/projected/c5aefada-2f47-4798-9b15-950327db9222-kube-api-access-kglrd\") pod \"authorino-57f7f8659f-5nrtq\" (UID: \"c5aefada-2f47-4798-9b15-950327db9222\") " pod="kuadrant-system/authorino-57f7f8659f-5nrtq" Oct 31 21:21:43.360318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.360249 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kglrd\" (UniqueName: \"kubernetes.io/projected/c5aefada-2f47-4798-9b15-950327db9222-kube-api-access-kglrd\") pod \"authorino-57f7f8659f-5nrtq\" (UID: \"c5aefada-2f47-4798-9b15-950327db9222\") " pod="kuadrant-system/authorino-57f7f8659f-5nrtq" Oct 31 21:21:43.458496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.458403 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" Oct 31 21:21:43.756706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:43.756680 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5nrtq"] Oct 31 21:21:43.758721 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:21:43.758689 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5aefada_2f47_4798_9b15_950327db9222.slice/crio-a7143246e87c16bf51050052a1f6666c2564823426c7c25c8aa8e1230f9f344a WatchSource:0}: Error finding container a7143246e87c16bf51050052a1f6666c2564823426c7c25c8aa8e1230f9f344a: Status 404 returned error can't find the container with id a7143246e87c16bf51050052a1f6666c2564823426c7c25c8aa8e1230f9f344a Oct 31 21:21:44.242706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:44.242675 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-cxhcb" Oct 31 21:21:44.298249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:44.298212 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" event={"ID":"c5aefada-2f47-4798-9b15-950327db9222","Type":"ContainerStarted","Data":"a7143246e87c16bf51050052a1f6666c2564823426c7c25c8aa8e1230f9f344a"} Oct 31 21:21:45.278560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:45.277853 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-s6zvf" Oct 31 21:21:46.311559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:46.311515 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" event={"ID":"c5aefada-2f47-4798-9b15-950327db9222","Type":"ContainerStarted","Data":"995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c"} Oct 31 21:21:46.352362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:46.352294 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" podStartSLOduration=1.7284905259999999 podStartE2EDuration="3.35227903s" totalImagesPullingTime="1.623788504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:46.351226877 +0000 UTC m=+353.044184270" watchObservedRunningTime="2026-10-31 21:21:46.35227903 +0000 UTC m=+353.045236474" Oct 31 21:21:46.428243 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:46.428200 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5nrtq"] Oct 31 21:21:48.321663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.321617 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" podUID="c5aefada-2f47-4798-9b15-950327db9222" containerName="authorino" containerID="cri-o://995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c" gracePeriod=30 Oct 31 21:21:48.571875 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.571810 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" Oct 31 21:21:48.684570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.684533 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5aefada-2f47-4798-9b15-950327db9222-kube-api-access-kglrd\" (UniqueName: \"kubernetes.io/projected/c5aefada-2f47-4798-9b15-950327db9222-kube-api-access-kglrd\") pod \"c5aefada-2f47-4798-9b15-950327db9222\" (UID: \"c5aefada-2f47-4798-9b15-950327db9222\") " Oct 31 21:21:48.687155 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.687116 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5aefada-2f47-4798-9b15-950327db9222-kube-api-access-kglrd" pod "c5aefada-2f47-4798-9b15-950327db9222" (UID: "c5aefada-2f47-4798-9b15-950327db9222"). InnerVolumeSpecName "kube-api-access-kglrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:21:48.785706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.785670 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kglrd\" (UniqueName: \"kubernetes.io/projected/c5aefada-2f47-4798-9b15-950327db9222-kube-api-access-kglrd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:21:48.938780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.938740 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9"] Oct 31 21:21:48.939211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.939192 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c5aefada-2f47-4798-9b15-950327db9222" containerName="authorino" Oct 31 21:21:48.939262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.939212 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5aefada-2f47-4798-9b15-950327db9222" containerName="authorino" Oct 31 21:21:48.939302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.939292 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5aefada-2f47-4798-9b15-950327db9222" containerName="authorino" Oct 31 21:21:48.985861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.985823 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9"] Oct 31 21:21:48.985861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:48.985845 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.008158 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.008121 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-jqrc8\"" Oct 31 21:21:49.008158 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.008151 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 21:21:49.008403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.008203 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 21:21:49.028267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.028236 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 21:21:49.087628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.087589 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.087816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.087651 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.087816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.087758 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnlz\" (UniqueName: \"kubernetes.io/projected/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-kube-api-access-rwnlz\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.087816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.087806 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.188616 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.188568 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.188834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.188640 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnlz\" (UniqueName: \"kubernetes.io/projected/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-kube-api-access-rwnlz\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.188834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.188673 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.188834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.188709 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.189058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.189040 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.189162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.189139 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.189539 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.189521 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.210436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.210402 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnlz\" (UniqueName: \"kubernetes.io/projected/ae34d5ae-7e14-4a0b-b89b-d22c254f7618-kube-api-access-rwnlz\") pod \"kserve-module-controller-manager-7566cf5fc8-j48f9\" (UID: \"ae34d5ae-7e14-4a0b-b89b-d22c254f7618\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.311056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.311013 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:49.327807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.327738 5188 generic.go:356] "Generic (PLEG): container finished" podID="c5aefada-2f47-4798-9b15-950327db9222" containerID="995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c" exitCode=0 Oct 31 21:21:49.327807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.327779 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" Oct 31 21:21:49.328125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.327814 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" event={"ID":"c5aefada-2f47-4798-9b15-950327db9222","Type":"ContainerDied","Data":"995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c"} Oct 31 21:21:49.328125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.327857 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-5nrtq" event={"ID":"c5aefada-2f47-4798-9b15-950327db9222","Type":"ContainerDied","Data":"a7143246e87c16bf51050052a1f6666c2564823426c7c25c8aa8e1230f9f344a"} Oct 31 21:21:49.328125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.327874 5188 scope.go:122] "RemoveContainer" containerID="995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c" Oct 31 21:21:49.338588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.338561 5188 scope.go:122] "RemoveContainer" containerID="995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c" Oct 31 21:21:49.338915 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:21:49.338895 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c\": container with ID starting with 995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c not found: ID does not exist" containerID="995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c" Oct 31 21:21:49.338992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.338924 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c"} err="failed to get container status \"995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c\": rpc error: code = NotFound desc = could not find container \"995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c\": container with ID starting with 995df6e5d6601bdb0268cd7c27e2dba75c82795777423653a9839e3112f8db5c not found: ID does not exist" Oct 31 21:21:49.392885 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.392778 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5nrtq"] Oct 31 21:21:49.398050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.398017 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-5nrtq"] Oct 31 21:21:49.471690 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.471600 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5aefada-2f47-4798-9b15-950327db9222" path="/var/lib/kubelet/pods/c5aefada-2f47-4798-9b15-950327db9222/volumes" Oct 31 21:21:49.642254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:49.642217 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9"] Oct 31 21:21:49.644365 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:21:49.644318 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae34d5ae_7e14_4a0b_b89b_d22c254f7618.slice/crio-f6eea3775887e50100eb3b657da3a07d4dc309e96e5deabec23cab58eba78057 WatchSource:0}: Error finding container f6eea3775887e50100eb3b657da3a07d4dc309e96e5deabec23cab58eba78057: Status 404 returned error can't find the container with id f6eea3775887e50100eb3b657da3a07d4dc309e96e5deabec23cab58eba78057 Oct 31 21:21:50.334623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:50.334578 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" event={"ID":"ae34d5ae-7e14-4a0b-b89b-d22c254f7618","Type":"ContainerStarted","Data":"f6eea3775887e50100eb3b657da3a07d4dc309e96e5deabec23cab58eba78057"} Oct 31 21:21:51.343611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:51.343576 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" event={"ID":"ae34d5ae-7e14-4a0b-b89b-d22c254f7618","Type":"ContainerStarted","Data":"68e1887deb67923ec38a95e6e101a875a1843a091c0f67f93b5878c9f0188fc7"} Oct 31 21:21:51.344044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:51.343703 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:21:51.382736 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:21:51.382674 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" podStartSLOduration=1.931674968 podStartE2EDuration="3.382657835s" totalImagesPullingTime="1.450982867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:51.382234768 +0000 UTC m=+358.075192161" watchObservedRunningTime="2026-10-31 21:21:51.382657835 +0000 UTC m=+358.075615402" Oct 31 21:22:02.535163 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:22:02.535122 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:22:12.350243 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:12.350207 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-j48f9" Oct 31 21:22:20.513551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.513508 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5cd76fccff-vhrnn"] Oct 31 21:22:20.522087 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.522052 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.536984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.536950 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 21:22:20.539272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.539236 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd76fccff-vhrnn"] Oct 31 21:22:20.546988 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.546955 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 21:22:20.547148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.547011 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 21:22:20.547148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.547067 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5ftdk\"" Oct 31 21:22:20.676712 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.676673 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-metrics-certs\") pod \"maas-controller-5cd76fccff-vhrnn\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.676909 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.676721 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lt5\" (UniqueName: \"kubernetes.io/projected/5772794a-a24a-4817-9dbc-a99353bc73c8-kube-api-access-42lt5\") pod \"maas-controller-5cd76fccff-vhrnn\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.676909 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.676831 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-webhook-cert\") pod \"maas-controller-5cd76fccff-vhrnn\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.778305 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.778202 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-metrics-certs\") pod \"maas-controller-5cd76fccff-vhrnn\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.778305 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.778246 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lt5\" (UniqueName: \"kubernetes.io/projected/5772794a-a24a-4817-9dbc-a99353bc73c8-kube-api-access-42lt5\") pod \"maas-controller-5cd76fccff-vhrnn\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.778560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.778315 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-webhook-cert\") pod \"maas-controller-5cd76fccff-vhrnn\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.781113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.781080 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-webhook-cert\") pod \"maas-controller-5cd76fccff-vhrnn\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.781288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.781080 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-metrics-certs\") pod \"maas-controller-5cd76fccff-vhrnn\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.799729 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.799689 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lt5\" (UniqueName: \"kubernetes.io/projected/5772794a-a24a-4817-9dbc-a99353bc73c8-kube-api-access-42lt5\") pod \"maas-controller-5cd76fccff-vhrnn\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:20.835549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:20.835511 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:21.002542 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.002509 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b9c46cb8-xlhtv"] Oct 31 21:22:21.007656 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.007631 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.033516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.033437 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9c46cb8-xlhtv"] Oct 31 21:22:21.136722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.136686 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5cd76fccff-vhrnn"] Oct 31 21:22:21.138886 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:22:21.138856 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5772794a_a24a_4817_9dbc_a99353bc73c8.slice/crio-e857c61552b195bfab91247871902c45588c0c457face1c5c55b3d928ed9f326 WatchSource:0}: Error finding container e857c61552b195bfab91247871902c45588c0c457face1c5c55b3d928ed9f326: Status 404 returned error can't find the container with id e857c61552b195bfab91247871902c45588c0c457face1c5c55b3d928ed9f326 Oct 31 21:22:21.182014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.181966 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d1ac43e-7514-4d51-947a-f3bf325f2a94-webhook-cert\") pod \"maas-controller-5b9c46cb8-xlhtv\" (UID: \"9d1ac43e-7514-4d51-947a-f3bf325f2a94\") " pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.182014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.182022 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1ac43e-7514-4d51-947a-f3bf325f2a94-metrics-certs\") pod \"maas-controller-5b9c46cb8-xlhtv\" (UID: \"9d1ac43e-7514-4d51-947a-f3bf325f2a94\") " pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.182251 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.182134 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhmq\" (UniqueName: \"kubernetes.io/projected/9d1ac43e-7514-4d51-947a-f3bf325f2a94-kube-api-access-9hhmq\") pod \"maas-controller-5b9c46cb8-xlhtv\" (UID: \"9d1ac43e-7514-4d51-947a-f3bf325f2a94\") " pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.283337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.283279 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhmq\" (UniqueName: \"kubernetes.io/projected/9d1ac43e-7514-4d51-947a-f3bf325f2a94-kube-api-access-9hhmq\") pod \"maas-controller-5b9c46cb8-xlhtv\" (UID: \"9d1ac43e-7514-4d51-947a-f3bf325f2a94\") " pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.283586 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.283405 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d1ac43e-7514-4d51-947a-f3bf325f2a94-webhook-cert\") pod \"maas-controller-5b9c46cb8-xlhtv\" (UID: \"9d1ac43e-7514-4d51-947a-f3bf325f2a94\") " pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.283586 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.283439 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1ac43e-7514-4d51-947a-f3bf325f2a94-metrics-certs\") pod \"maas-controller-5b9c46cb8-xlhtv\" (UID: \"9d1ac43e-7514-4d51-947a-f3bf325f2a94\") " pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.286062 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.286032 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9d1ac43e-7514-4d51-947a-f3bf325f2a94-webhook-cert\") pod \"maas-controller-5b9c46cb8-xlhtv\" (UID: \"9d1ac43e-7514-4d51-947a-f3bf325f2a94\") " pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.286180 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.286150 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d1ac43e-7514-4d51-947a-f3bf325f2a94-metrics-certs\") pod \"maas-controller-5b9c46cb8-xlhtv\" (UID: \"9d1ac43e-7514-4d51-947a-f3bf325f2a94\") " pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.305130 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.305095 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhmq\" (UniqueName: \"kubernetes.io/projected/9d1ac43e-7514-4d51-947a-f3bf325f2a94-kube-api-access-9hhmq\") pod \"maas-controller-5b9c46cb8-xlhtv\" (UID: \"9d1ac43e-7514-4d51-947a-f3bf325f2a94\") " pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.320935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.320904 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:21.500530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.500484 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" event={"ID":"5772794a-a24a-4817-9dbc-a99353bc73c8","Type":"ContainerStarted","Data":"e857c61552b195bfab91247871902c45588c0c457face1c5c55b3d928ed9f326"} Oct 31 21:22:21.619938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:21.619909 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b9c46cb8-xlhtv"] Oct 31 21:22:21.622200 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:22:21.622164 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1ac43e_7514_4d51_947a_f3bf325f2a94.slice/crio-bbe178204899ea2cc2776357a4e5a05aa12adb091d93912c9ddd050ff0b4628f WatchSource:0}: Error finding container bbe178204899ea2cc2776357a4e5a05aa12adb091d93912c9ddd050ff0b4628f: Status 404 returned error can't find the container with id bbe178204899ea2cc2776357a4e5a05aa12adb091d93912c9ddd050ff0b4628f Oct 31 21:22:22.512546 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:22.512499 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" event={"ID":"9d1ac43e-7514-4d51-947a-f3bf325f2a94","Type":"ContainerStarted","Data":"bbe178204899ea2cc2776357a4e5a05aa12adb091d93912c9ddd050ff0b4628f"} Oct 31 21:22:24.533582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:24.533482 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" event={"ID":"5772794a-a24a-4817-9dbc-a99353bc73c8","Type":"ContainerStarted","Data":"0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8"} Oct 31 21:22:24.534025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:24.533607 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:24.535258 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:24.535231 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" event={"ID":"9d1ac43e-7514-4d51-947a-f3bf325f2a94","Type":"ContainerStarted","Data":"b4ac7ccb4aae8fd120542b386f5792354e5d3ef80976ba0469ed4b1ab8e8efe3"} Oct 31 21:22:24.535388 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:24.535296 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:24.573955 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:24.573882 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" podStartSLOduration=1.76259628 podStartE2EDuration="4.573862421s" totalImagesPullingTime="2.811266141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:22:24.572031337 +0000 UTC m=+391.264988755" watchObservedRunningTime="2026-10-31 21:22:24.573862421 +0000 UTC m=+391.266819825" Oct 31 21:22:24.618786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:24.618715 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" podStartSLOduration=2.02246729 podStartE2EDuration="4.618700018s" totalImagesPullingTime="2.596232728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:22:24.616889841 +0000 UTC m=+391.309847232" watchObservedRunningTime="2026-10-31 21:22:24.618700018 +0000 UTC m=+391.311657410" Oct 31 21:22:25.786937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.786896 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4"] Oct 31 21:22:25.793979 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.793945 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:25.811826 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.811789 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4"] Oct 31 21:22:25.818772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.818742 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 21:22:25.818910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.818855 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 21:22:25.818980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.818948 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-cdtpf\"" Oct 31 21:22:25.819026 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.819000 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 21:22:25.926507 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.926465 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gnx\" (UniqueName: \"kubernetes.io/projected/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-kube-api-access-s4gnx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:25.926685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.926534 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:25.926685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.926590 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:25.926685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:25.926628 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.027764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.027722 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gnx\" (UniqueName: \"kubernetes.io/projected/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-kube-api-access-s4gnx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.027935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.027786 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.027935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.027815 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.027935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.027842 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.028516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.028490 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.028748 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.028729 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.030176 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.030151 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.049360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.049261 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gnx\" (UniqueName: \"kubernetes.io/projected/335b5cbb-fd40-44fd-9d2f-ec66c41cae34-kube-api-access-s4gnx\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hbkp4\" (UID: \"335b5cbb-fd40-44fd-9d2f-ec66c41cae34\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.111882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.111829 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:26.412599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.412574 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4"] Oct 31 21:22:26.414580 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:22:26.414539 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335b5cbb_fd40_44fd_9d2f_ec66c41cae34.slice/crio-d2fd506ed4db173ca384a132a9921a8e84b6def451964bf7bf4d97622b889650 WatchSource:0}: Error finding container d2fd506ed4db173ca384a132a9921a8e84b6def451964bf7bf4d97622b889650: Status 404 returned error can't find the container with id d2fd506ed4db173ca384a132a9921a8e84b6def451964bf7bf4d97622b889650 Oct 31 21:22:26.546314 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:26.546272 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" event={"ID":"335b5cbb-fd40-44fd-9d2f-ec66c41cae34","Type":"ContainerStarted","Data":"d2fd506ed4db173ca384a132a9921a8e84b6def451964bf7bf4d97622b889650"} Oct 31 21:22:32.539287 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:22:32.539245 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:22:35.543167 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:35.543071 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:35.543676 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:35.543655 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:22:35.669628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:35.669586 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd76fccff-vhrnn"] Oct 31 21:22:35.669826 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:35.669796 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" podUID="5772794a-a24a-4817-9dbc-a99353bc73c8" containerName="manager" containerID="cri-o://0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8" gracePeriod=30 Oct 31 21:22:35.946576 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:35.946542 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:36.027571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.027538 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-metrics-certs\") pod \"5772794a-a24a-4817-9dbc-a99353bc73c8\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " Oct 31 21:22:36.027571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.027579 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-webhook-cert\") pod \"5772794a-a24a-4817-9dbc-a99353bc73c8\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " Oct 31 21:22:36.027829 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.027723 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5772794a-a24a-4817-9dbc-a99353bc73c8-kube-api-access-42lt5\" (UniqueName: \"kubernetes.io/projected/5772794a-a24a-4817-9dbc-a99353bc73c8-kube-api-access-42lt5\") pod \"5772794a-a24a-4817-9dbc-a99353bc73c8\" (UID: \"5772794a-a24a-4817-9dbc-a99353bc73c8\") " Oct 31 21:22:36.030051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.030010 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-webhook-cert" pod "5772794a-a24a-4817-9dbc-a99353bc73c8" (UID: "5772794a-a24a-4817-9dbc-a99353bc73c8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:22:36.030194 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.030164 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-metrics-certs" pod "5772794a-a24a-4817-9dbc-a99353bc73c8" (UID: "5772794a-a24a-4817-9dbc-a99353bc73c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:22:36.030255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.030208 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5772794a-a24a-4817-9dbc-a99353bc73c8-kube-api-access-42lt5" pod "5772794a-a24a-4817-9dbc-a99353bc73c8" (UID: "5772794a-a24a-4817-9dbc-a99353bc73c8"). InnerVolumeSpecName "kube-api-access-42lt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:22:36.128595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.128549 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:22:36.128595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.128588 5188 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5772794a-a24a-4817-9dbc-a99353bc73c8-webhook-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:22:36.128595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.128598 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42lt5\" (UniqueName: \"kubernetes.io/projected/5772794a-a24a-4817-9dbc-a99353bc73c8-kube-api-access-42lt5\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:22:36.621965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.621924 5188 generic.go:356] "Generic (PLEG): container finished" podID="5772794a-a24a-4817-9dbc-a99353bc73c8" containerID="0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8" exitCode=0 Oct 31 21:22:36.622395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.622111 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" event={"ID":"5772794a-a24a-4817-9dbc-a99353bc73c8","Type":"ContainerDied","Data":"0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8"} Oct 31 21:22:36.622395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.622154 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" event={"ID":"5772794a-a24a-4817-9dbc-a99353bc73c8","Type":"ContainerDied","Data":"e857c61552b195bfab91247871902c45588c0c457face1c5c55b3d928ed9f326"} Oct 31 21:22:36.622395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.622177 5188 scope.go:122] "RemoveContainer" containerID="0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8" Oct 31 21:22:36.622395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.622338 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5cd76fccff-vhrnn" Oct 31 21:22:36.634398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.634374 5188 scope.go:122] "RemoveContainer" containerID="0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8" Oct 31 21:22:36.634787 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:22:36.634757 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8\": container with ID starting with 0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8 not found: ID does not exist" containerID="0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8" Oct 31 21:22:36.634889 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.634793 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8"} err="failed to get container status \"0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8\": rpc error: code = NotFound desc = could not find container \"0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8\": container with ID starting with 0090deb802a83afe89fc1e295f57cc969eafac194d4e632542fefb750219b9b8 not found: ID does not exist" Oct 31 21:22:36.667880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.667848 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5cd76fccff-vhrnn"] Oct 31 21:22:36.677617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:36.677577 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5cd76fccff-vhrnn"] Oct 31 21:22:37.471418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:37.471381 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5772794a-a24a-4817-9dbc-a99353bc73c8" path="/var/lib/kubelet/pods/5772794a-a24a-4817-9dbc-a99353bc73c8/volumes" Oct 31 21:22:37.628448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:37.628409 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" event={"ID":"335b5cbb-fd40-44fd-9d2f-ec66c41cae34","Type":"ContainerStarted","Data":"4c2603bc5ce22a4398b71a76ad27b6dd8781f9f7d46b24a577ee240bf68e9a49"} Oct 31 21:22:37.628918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:37.628532 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:37.666633 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:37.666574 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" podStartSLOduration=2.149782771 podStartE2EDuration="12.666559488s" totalImagesPullingTime="10.516776717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:22:37.665530719 +0000 UTC m=+404.358488112" watchObservedRunningTime="2026-10-31 21:22:37.666559488 +0000 UTC m=+404.359516879" Oct 31 21:22:47.271249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.271210 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-zpt8v"] Oct 31 21:22:47.271902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.271849 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5772794a-a24a-4817-9dbc-a99353bc73c8" containerName="manager" Oct 31 21:22:47.271902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.271870 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5772794a-a24a-4817-9dbc-a99353bc73c8" containerName="manager" Oct 31 21:22:47.272048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.271951 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5772794a-a24a-4817-9dbc-a99353bc73c8" containerName="manager" Oct 31 21:22:47.278965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.278934 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:22:47.305420 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.305393 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 21:22:47.305594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.305431 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tjvcp\"" Oct 31 21:22:47.309791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.309765 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-zpt8v"] Oct 31 21:22:47.334899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.334866 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6142fdd-301b-49dc-aa47-689dcfe7c061-cert\") pod \"kserve-controller-manager-776cc66d47-zpt8v\" (UID: \"c6142fdd-301b-49dc-aa47-689dcfe7c061\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:22:47.335070 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.334925 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjx2k\" (UniqueName: \"kubernetes.io/projected/c6142fdd-301b-49dc-aa47-689dcfe7c061-kube-api-access-rjx2k\") pod \"kserve-controller-manager-776cc66d47-zpt8v\" (UID: \"c6142fdd-301b-49dc-aa47-689dcfe7c061\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:22:47.419762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.419729 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-25bhl"] Oct 31 21:22:47.423407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.423382 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:22:47.435601 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.435574 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjx2k\" (UniqueName: \"kubernetes.io/projected/c6142fdd-301b-49dc-aa47-689dcfe7c061-kube-api-access-rjx2k\") pod \"kserve-controller-manager-776cc66d47-zpt8v\" (UID: \"c6142fdd-301b-49dc-aa47-689dcfe7c061\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:22:47.435729 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.435675 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6142fdd-301b-49dc-aa47-689dcfe7c061-cert\") pod \"kserve-controller-manager-776cc66d47-zpt8v\" (UID: \"c6142fdd-301b-49dc-aa47-689dcfe7c061\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:22:47.438018 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.437994 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6142fdd-301b-49dc-aa47-689dcfe7c061-cert\") pod \"kserve-controller-manager-776cc66d47-zpt8v\" (UID: \"c6142fdd-301b-49dc-aa47-689dcfe7c061\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:22:47.449472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.449442 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-25bhl"] Oct 31 21:22:47.473078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.473044 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjx2k\" (UniqueName: \"kubernetes.io/projected/c6142fdd-301b-49dc-aa47-689dcfe7c061-kube-api-access-rjx2k\") pod \"kserve-controller-manager-776cc66d47-zpt8v\" (UID: \"c6142fdd-301b-49dc-aa47-689dcfe7c061\") " pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:22:47.485207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.485178 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-5wlcw"] Oct 31 21:22:47.488944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.488926 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.497510 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.497482 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-9pfrl\"" Oct 31 21:22:47.497603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.497506 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 21:22:47.502645 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.502627 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-s5qqt\"" Oct 31 21:22:47.502756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.502713 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 21:22:47.502756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.502733 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 21:22:47.518915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.514062 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5wlcw"] Oct 31 21:22:47.536666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.536574 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frt2z\" (UniqueName: \"kubernetes.io/projected/ed5c581e-3188-4ecf-acb4-6c0fc04dc146-kube-api-access-frt2z\") pod \"model-serving-api-75bd89fdfc-25bhl\" (UID: \"ed5c581e-3188-4ecf-acb4-6c0fc04dc146\") " pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:22:47.536666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.536638 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlg29\" (UniqueName: \"kubernetes.io/projected/4a78bc84-6872-4292-b6f5-60e3737194a7-kube-api-access-xlg29\") pod \"odh-model-controller-769875db56-5wlcw\" (UID: \"4a78bc84-6872-4292-b6f5-60e3737194a7\") " pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.536837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.536750 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a78bc84-6872-4292-b6f5-60e3737194a7-cert\") pod \"odh-model-controller-769875db56-5wlcw\" (UID: \"4a78bc84-6872-4292-b6f5-60e3737194a7\") " pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.536837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.536799 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5c581e-3188-4ecf-acb4-6c0fc04dc146-tls-certs\") pod \"model-serving-api-75bd89fdfc-25bhl\" (UID: \"ed5c581e-3188-4ecf-acb4-6c0fc04dc146\") " pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:22:47.536927 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.536905 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a78bc84-6872-4292-b6f5-60e3737194a7-metrics-certs\") pod \"odh-model-controller-769875db56-5wlcw\" (UID: \"4a78bc84-6872-4292-b6f5-60e3737194a7\") " pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.592508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.592468 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:22:47.637829 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.637789 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a78bc84-6872-4292-b6f5-60e3737194a7-metrics-certs\") pod \"odh-model-controller-769875db56-5wlcw\" (UID: \"4a78bc84-6872-4292-b6f5-60e3737194a7\") " pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.638008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.637839 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt2z\" (UniqueName: \"kubernetes.io/projected/ed5c581e-3188-4ecf-acb4-6c0fc04dc146-kube-api-access-frt2z\") pod \"model-serving-api-75bd89fdfc-25bhl\" (UID: \"ed5c581e-3188-4ecf-acb4-6c0fc04dc146\") " pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:22:47.638008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.637876 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlg29\" (UniqueName: \"kubernetes.io/projected/4a78bc84-6872-4292-b6f5-60e3737194a7-kube-api-access-xlg29\") pod \"odh-model-controller-769875db56-5wlcw\" (UID: \"4a78bc84-6872-4292-b6f5-60e3737194a7\") " pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.638008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.637900 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a78bc84-6872-4292-b6f5-60e3737194a7-cert\") pod \"odh-model-controller-769875db56-5wlcw\" (UID: \"4a78bc84-6872-4292-b6f5-60e3737194a7\") " pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.638008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.637943 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5c581e-3188-4ecf-acb4-6c0fc04dc146-tls-certs\") pod \"model-serving-api-75bd89fdfc-25bhl\" (UID: \"ed5c581e-3188-4ecf-acb4-6c0fc04dc146\") " pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:22:47.641037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.640981 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a78bc84-6872-4292-b6f5-60e3737194a7-cert\") pod \"odh-model-controller-769875db56-5wlcw\" (UID: \"4a78bc84-6872-4292-b6f5-60e3737194a7\") " pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.641230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.641116 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a78bc84-6872-4292-b6f5-60e3737194a7-metrics-certs\") pod \"odh-model-controller-769875db56-5wlcw\" (UID: \"4a78bc84-6872-4292-b6f5-60e3737194a7\") " pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.641468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.641447 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed5c581e-3188-4ecf-acb4-6c0fc04dc146-tls-certs\") pod \"model-serving-api-75bd89fdfc-25bhl\" (UID: \"ed5c581e-3188-4ecf-acb4-6c0fc04dc146\") " pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:22:47.660534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.660496 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlg29\" (UniqueName: \"kubernetes.io/projected/4a78bc84-6872-4292-b6f5-60e3737194a7-kube-api-access-xlg29\") pod \"odh-model-controller-769875db56-5wlcw\" (UID: \"4a78bc84-6872-4292-b6f5-60e3737194a7\") " pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.661691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.661490 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frt2z\" (UniqueName: \"kubernetes.io/projected/ed5c581e-3188-4ecf-acb4-6c0fc04dc146-kube-api-access-frt2z\") pod \"model-serving-api-75bd89fdfc-25bhl\" (UID: \"ed5c581e-3188-4ecf-acb4-6c0fc04dc146\") " pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:22:47.736372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.736315 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:22:47.802814 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.802777 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:47.892984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:47.892956 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-zpt8v"] Oct 31 21:22:47.895146 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:22:47.895108 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6142fdd_301b_49dc_aa47_689dcfe7c061.slice/crio-6c20d5754d4c5e42c0a22df9128144ee4771e8b1e2e0ba3f7aa345f52d0f52f6 WatchSource:0}: Error finding container 6c20d5754d4c5e42c0a22df9128144ee4771e8b1e2e0ba3f7aa345f52d0f52f6: Status 404 returned error can't find the container with id 6c20d5754d4c5e42c0a22df9128144ee4771e8b1e2e0ba3f7aa345f52d0f52f6 Oct 31 21:22:48.036161 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:48.036130 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-25bhl"] Oct 31 21:22:48.036995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:22:48.036968 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5c581e_3188_4ecf_acb4_6c0fc04dc146.slice/crio-0e788d78f556053191eea15bd3f384cea3034d1e79d8bf02df41c5a978ae6897 WatchSource:0}: Error finding container 0e788d78f556053191eea15bd3f384cea3034d1e79d8bf02df41c5a978ae6897: Status 404 returned error can't find the container with id 0e788d78f556053191eea15bd3f384cea3034d1e79d8bf02df41c5a978ae6897 Oct 31 21:22:48.106913 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:48.106888 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-5wlcw"] Oct 31 21:22:48.108409 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:22:48.108381 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a78bc84_6872_4292_b6f5_60e3737194a7.slice/crio-80c0af51d3bd7855d7d53ff3dd90e8cef5b515d382177e090085340d8d67cec7 WatchSource:0}: Error finding container 80c0af51d3bd7855d7d53ff3dd90e8cef5b515d382177e090085340d8d67cec7: Status 404 returned error can't find the container with id 80c0af51d3bd7855d7d53ff3dd90e8cef5b515d382177e090085340d8d67cec7 Oct 31 21:22:48.644052 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:48.644018 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hbkp4" Oct 31 21:22:48.727803 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:48.727760 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5wlcw" event={"ID":"4a78bc84-6872-4292-b6f5-60e3737194a7","Type":"ContainerStarted","Data":"80c0af51d3bd7855d7d53ff3dd90e8cef5b515d382177e090085340d8d67cec7"} Oct 31 21:22:48.732260 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:48.732152 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" event={"ID":"c6142fdd-301b-49dc-aa47-689dcfe7c061","Type":"ContainerStarted","Data":"6c20d5754d4c5e42c0a22df9128144ee4771e8b1e2e0ba3f7aa345f52d0f52f6"} Oct 31 21:22:48.738313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:48.738268 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" event={"ID":"ed5c581e-3188-4ecf-acb4-6c0fc04dc146","Type":"ContainerStarted","Data":"0e788d78f556053191eea15bd3f384cea3034d1e79d8bf02df41c5a978ae6897"} Oct 31 21:22:50.754882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:50.754785 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" event={"ID":"c6142fdd-301b-49dc-aa47-689dcfe7c061","Type":"ContainerStarted","Data":"5022943ab41658b5700cde62c71478d78cc274d26f57eaccbb4502263ed5ef30"} Oct 31 21:22:50.755351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:50.755081 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:22:50.756963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:50.756930 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" event={"ID":"ed5c581e-3188-4ecf-acb4-6c0fc04dc146","Type":"ContainerStarted","Data":"61e5acffae116a59f768cf9e5eac9ad8fd2bc54c9a1668e5fc1f1022d790ac90"} Oct 31 21:22:50.757104 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:50.757013 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:22:50.792828 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:50.792765 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" podStartSLOduration=1.229357847 podStartE2EDuration="3.792744917s" totalImagesPullingTime="2.56338707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:22:50.791209195 +0000 UTC m=+417.484166588" watchObservedRunningTime="2026-10-31 21:22:50.792744917 +0000 UTC m=+417.485702312" Oct 31 21:22:50.837122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:50.837059 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" podStartSLOduration=1.680711843 podStartE2EDuration="3.837045088s" totalImagesPullingTime="2.156333245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:22:50.835580821 +0000 UTC m=+417.528538213" watchObservedRunningTime="2026-10-31 21:22:50.837045088 +0000 UTC m=+417.530002484" Oct 31 21:22:51.764399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:51.764349 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-5wlcw" event={"ID":"4a78bc84-6872-4292-b6f5-60e3737194a7","Type":"ContainerStarted","Data":"1a46df648ae1d1fb0bebee843ba5e94fd42cfd1def01f5476ae49708e74e19b3"} Oct 31 21:22:51.764884 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:51.764614 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:22:51.803166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:22:51.803104 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-5wlcw" podStartSLOduration=1.853460688 podStartE2EDuration="4.803089128s" totalImagesPullingTime="2.94962844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:22:51.800915979 +0000 UTC m=+418.493873371" watchObservedRunningTime="2026-10-31 21:22:51.803089128 +0000 UTC m=+418.496046534" Oct 31 21:23:01.769080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:01.769046 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-25bhl" Oct 31 21:23:02.536105 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:23:02.536060 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:23:02.771418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:02.771385 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-5wlcw" Oct 31 21:23:04.460653 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.460613 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rgwv5"] Oct 31 21:23:04.834491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.834400 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rgwv5"] Oct 31 21:23:04.834491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.834461 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rgwv5"] Oct 31 21:23:04.834696 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.834533 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rgwv5" Oct 31 21:23:04.834985 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:23:04.834957 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-sf5ql], unattached volumes=[], failed to process volumes=[kube-api-access-sf5ql]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-rgwv5" podUID="102e9315-9aa9-4ba5-b53e-c3c0510b48f5" Oct 31 21:23:04.841470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.841438 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rgwv5" Oct 31 21:23:04.877642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.877609 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rgwv5" Oct 31 21:23:04.933715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.933678 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8687975cdd-6l8tp"] Oct 31 21:23:04.940797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.940764 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:23:04.956043 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.956002 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-956cc\"" Oct 31 21:23:04.956206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.956102 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 21:23:04.958498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:04.958467 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8687975cdd-6l8tp"] Oct 31 21:23:05.003974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.003937 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1a40dbf-9add-4726-b0e4-da271ad584f4-tls-cert\") pod \"authorino-8687975cdd-6l8tp\" (UID: \"f1a40dbf-9add-4726-b0e4-da271ad584f4\") " pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:23:05.004148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.003981 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qsp4\" (UniqueName: \"kubernetes.io/projected/f1a40dbf-9add-4726-b0e4-da271ad584f4-kube-api-access-6qsp4\") pod \"authorino-8687975cdd-6l8tp\" (UID: \"f1a40dbf-9add-4726-b0e4-da271ad584f4\") " pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:23:05.105346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.105247 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qsp4\" (UniqueName: \"kubernetes.io/projected/f1a40dbf-9add-4726-b0e4-da271ad584f4-kube-api-access-6qsp4\") pod \"authorino-8687975cdd-6l8tp\" (UID: \"f1a40dbf-9add-4726-b0e4-da271ad584f4\") " pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:23:05.105518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.105373 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1a40dbf-9add-4726-b0e4-da271ad584f4-tls-cert\") pod \"authorino-8687975cdd-6l8tp\" (UID: \"f1a40dbf-9add-4726-b0e4-da271ad584f4\") " pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:23:05.107892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.107861 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1a40dbf-9add-4726-b0e4-da271ad584f4-tls-cert\") pod \"authorino-8687975cdd-6l8tp\" (UID: \"f1a40dbf-9add-4726-b0e4-da271ad584f4\") " pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:23:05.126931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.126887 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qsp4\" (UniqueName: \"kubernetes.io/projected/f1a40dbf-9add-4726-b0e4-da271ad584f4-kube-api-access-6qsp4\") pod \"authorino-8687975cdd-6l8tp\" (UID: \"f1a40dbf-9add-4726-b0e4-da271ad584f4\") " pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:23:05.255516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.255472 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:23:05.572100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.572071 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8687975cdd-6l8tp"] Oct 31 21:23:05.573939 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:23:05.573912 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a40dbf_9add_4726_b0e4_da271ad584f4.slice/crio-8f60904e054e38795c486ab55c5b812a67a6d2d8d5b6375f75dc890f4e876f1c WatchSource:0}: Error finding container 8f60904e054e38795c486ab55c5b812a67a6d2d8d5b6375f75dc890f4e876f1c: Status 404 returned error can't find the container with id 8f60904e054e38795c486ab55c5b812a67a6d2d8d5b6375f75dc890f4e876f1c Oct 31 21:23:05.848522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.848414 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8687975cdd-6l8tp" event={"ID":"f1a40dbf-9add-4726-b0e4-da271ad584f4","Type":"ContainerStarted","Data":"8f60904e054e38795c486ab55c5b812a67a6d2d8d5b6375f75dc890f4e876f1c"} Oct 31 21:23:05.848522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.848505 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rgwv5" Oct 31 21:23:05.913936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.913898 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rgwv5"] Oct 31 21:23:05.921065 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:05.921027 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rgwv5"] Oct 31 21:23:06.856816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:06.856710 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8687975cdd-6l8tp" event={"ID":"f1a40dbf-9add-4726-b0e4-da271ad584f4","Type":"ContainerStarted","Data":"bfc3ab26f927d8d12b086e2ab35e4473a89385143b2a5424e55930131720739e"} Oct 31 21:23:06.894850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:06.892834 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-8687975cdd-6l8tp" podStartSLOduration=2.020388637 podStartE2EDuration="2.892813937s" totalImagesPullingTime="872.4253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:23:06.891609643 +0000 UTC m=+433.584567047" watchObservedRunningTime="2026-10-31 21:23:06.892813937 +0000 UTC m=+433.585771331" Oct 31 21:23:07.471426 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:07.471387 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="102e9315-9aa9-4ba5-b53e-c3c0510b48f5" path="/var/lib/kubelet/pods/102e9315-9aa9-4ba5-b53e-c3c0510b48f5/volumes" Oct 31 21:23:21.770263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:23:21.770232 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-zpt8v" Oct 31 21:23:32.535962 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:23:32.535922 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:24:02.536621 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:24:02.536580 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:24:26.470777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.470665 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-f9sls"] Oct 31 21:24:26.474597 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.474570 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:26.494531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.494497 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-f9sls"] Oct 31 21:24:26.521524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.521494 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-627x2\"" Oct 31 21:24:26.527754 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.527730 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 21:24:26.542195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.542160 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe61268d-fbb8-4676-80cf-0309e133856e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-f9sls\" (UID: \"fe61268d-fbb8-4676-80cf-0309e133856e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:26.542364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.542290 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj89n\" (UniqueName: \"kubernetes.io/projected/fe61268d-fbb8-4676-80cf-0309e133856e-kube-api-access-bj89n\") pod \"payload-pre-processing-7b98bb678c-f9sls\" (UID: \"fe61268d-fbb8-4676-80cf-0309e133856e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:26.643435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.643400 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj89n\" (UniqueName: \"kubernetes.io/projected/fe61268d-fbb8-4676-80cf-0309e133856e-kube-api-access-bj89n\") pod \"payload-pre-processing-7b98bb678c-f9sls\" (UID: \"fe61268d-fbb8-4676-80cf-0309e133856e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:26.643655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.643460 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe61268d-fbb8-4676-80cf-0309e133856e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-f9sls\" (UID: \"fe61268d-fbb8-4676-80cf-0309e133856e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:26.644113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.644090 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe61268d-fbb8-4676-80cf-0309e133856e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-f9sls\" (UID: \"fe61268d-fbb8-4676-80cf-0309e133856e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:26.664478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.664437 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj89n\" (UniqueName: \"kubernetes.io/projected/fe61268d-fbb8-4676-80cf-0309e133856e-kube-api-access-bj89n\") pod \"payload-pre-processing-7b98bb678c-f9sls\" (UID: \"fe61268d-fbb8-4676-80cf-0309e133856e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:26.788049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:26.787955 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:27.086879 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:27.086850 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-f9sls"] Oct 31 21:24:27.089201 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:24:27.089163 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe61268d_fbb8_4676_80cf_0309e133856e.slice/crio-c9ad9e2c2f7ad0d362962669a2d3e749c09301c6adad22b6a2e75afe1b4dc3e6 WatchSource:0}: Error finding container c9ad9e2c2f7ad0d362962669a2d3e749c09301c6adad22b6a2e75afe1b4dc3e6: Status 404 returned error can't find the container with id c9ad9e2c2f7ad0d362962669a2d3e749c09301c6adad22b6a2e75afe1b4dc3e6 Oct 31 21:24:27.261434 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:27.261394 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" event={"ID":"fe61268d-fbb8-4676-80cf-0309e133856e","Type":"ContainerStarted","Data":"c9ad9e2c2f7ad0d362962669a2d3e749c09301c6adad22b6a2e75afe1b4dc3e6"} Oct 31 21:24:29.272892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:29.272854 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" event={"ID":"fe61268d-fbb8-4676-80cf-0309e133856e","Type":"ContainerStarted","Data":"4b96165ad9fde5612eb546e3fe57fd7943c456cf94e0d5d7437c14bfc5b5ef82"} Oct 31 21:24:29.273275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:29.272996 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:29.308745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:29.308691 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" podStartSLOduration=1.283935099 podStartE2EDuration="3.308677372s" totalImagesPullingTime="2.024742273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:24:29.308114162 +0000 UTC m=+516.001071568" watchObservedRunningTime="2026-10-31 21:24:29.308677372 +0000 UTC m=+516.001634764" Oct 31 21:24:32.535728 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:24:32.535690 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:24:40.282316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:40.282283 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-f9sls" Oct 31 21:24:51.390498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:51.390458 5188 generic.go:356] "Generic (PLEG): container finished" podID="9d1ac43e-7514-4d51-947a-f3bf325f2a94" containerID="b4ac7ccb4aae8fd120542b386f5792354e5d3ef80976ba0469ed4b1ab8e8efe3" exitCode=1 Oct 31 21:24:51.390901 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:51.390535 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" event={"ID":"9d1ac43e-7514-4d51-947a-f3bf325f2a94","Type":"ContainerDied","Data":"b4ac7ccb4aae8fd120542b386f5792354e5d3ef80976ba0469ed4b1ab8e8efe3"} Oct 31 21:24:51.390945 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:51.390913 5188 scope.go:122] "RemoveContainer" containerID="b4ac7ccb4aae8fd120542b386f5792354e5d3ef80976ba0469ed4b1ab8e8efe3" Oct 31 21:24:52.397292 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:52.397251 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" event={"ID":"9d1ac43e-7514-4d51-947a-f3bf325f2a94","Type":"ContainerStarted","Data":"0e3b7cd6a757a659171dc2cb25229c245f477c33f775c084ff83e330ace66fc8"} Oct 31 21:24:52.397684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:24:52.397488 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:25:02.536850 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:25:02.536806 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:25:03.404903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:03.404871 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b9c46cb8-xlhtv" Oct 31 21:25:08.726994 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.726942 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls"] Oct 31 21:25:08.731278 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.731251 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.744026 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.744003 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 21:25:08.753598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.753566 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 21:25:08.753756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.753647 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 21:25:08.753756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.753699 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-r4s2n\"" Oct 31 21:25:08.756223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.756198 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls"] Oct 31 21:25:08.820471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.820431 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.820649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.820492 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9npq\" (UniqueName: \"kubernetes.io/projected/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-kube-api-access-v9npq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.820649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.820594 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.820649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.820618 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.820649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.820636 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.820775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.820653 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.921827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.921787 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.921827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.921829 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.922050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.921855 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.922050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.921969 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.922157 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.922054 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.922157 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.922092 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9npq\" (UniqueName: \"kubernetes.io/projected/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-kube-api-access-v9npq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.922269 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.922251 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.922366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.922317 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.922448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.922427 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.924140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.924122 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.924446 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.924425 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:08.943743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:08.943711 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9npq\" (UniqueName: \"kubernetes.io/projected/8fae092c-44ce-4bc7-9c06-5f946eee3d2a-kube-api-access-v9npq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-2trls\" (UID: \"8fae092c-44ce-4bc7-9c06-5f946eee3d2a\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:09.045403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:09.045289 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:09.356129 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:09.356102 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls"] Oct 31 21:25:09.358011 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:25:09.357980 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fae092c_44ce_4bc7_9c06_5f946eee3d2a.slice/crio-b49330f1b2eb31d0d794ab798e4237f6607fd3b115749ff1f0e43e3e1c279a1b WatchSource:0}: Error finding container b49330f1b2eb31d0d794ab798e4237f6607fd3b115749ff1f0e43e3e1c279a1b: Status 404 returned error can't find the container with id b49330f1b2eb31d0d794ab798e4237f6607fd3b115749ff1f0e43e3e1c279a1b Oct 31 21:25:09.498916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:09.498849 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" event={"ID":"8fae092c-44ce-4bc7-9c06-5f946eee3d2a","Type":"ContainerStarted","Data":"b49330f1b2eb31d0d794ab798e4237f6607fd3b115749ff1f0e43e3e1c279a1b"} Oct 31 21:25:11.512787 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:11.512751 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" event={"ID":"8fae092c-44ce-4bc7-9c06-5f946eee3d2a","Type":"ContainerStarted","Data":"9eeb2584621e21ac5cdccaf8ae92e38cc66aa1ffec7defb15abf6e978817a9ab"} Oct 31 21:25:11.552781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:11.552719 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" podStartSLOduration=2.251107673 podStartE2EDuration="3.552699138s" totalImagesPullingTime="1.301591465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:25:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:25:11.550417178 +0000 UTC m=+558.243374571" watchObservedRunningTime="2026-10-31 21:25:11.552699138 +0000 UTC m=+558.245656531" Oct 31 21:25:12.045888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:12.045850 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:18.434877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.434835 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf"] Oct 31 21:25:18.439568 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.439539 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.459566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.459538 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf"] Oct 31 21:25:18.462966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.462942 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 21:25:18.512807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.512765 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.512972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.512822 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c844161-8129-4828-9e4c-2cc46c64adca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.512972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.512938 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.512972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.512965 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.513075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.512985 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4qxv\" (UniqueName: \"kubernetes.io/projected/7c844161-8129-4828-9e4c-2cc46c64adca-kube-api-access-g4qxv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.513108 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.513094 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.614238 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.614187 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.614459 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.614283 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.614459 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.614353 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c844161-8129-4828-9e4c-2cc46c64adca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.614459 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.614423 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.614459 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.614454 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.614689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.614485 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4qxv\" (UniqueName: \"kubernetes.io/projected/7c844161-8129-4828-9e4c-2cc46c64adca-kube-api-access-g4qxv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.614748 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.614689 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.614891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.614863 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.615028 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.614894 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.616830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.616806 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c844161-8129-4828-9e4c-2cc46c64adca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.617198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.617175 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c844161-8129-4828-9e4c-2cc46c64adca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.638769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.638728 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4qxv\" (UniqueName: \"kubernetes.io/projected/7c844161-8129-4828-9e4c-2cc46c64adca-kube-api-access-g4qxv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf\" (UID: \"7c844161-8129-4828-9e4c-2cc46c64adca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:18.755834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:18.755732 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:19.046360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:19.046256 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:19.056551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:19.056529 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf"] Oct 31 21:25:19.058789 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:25:19.058762 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c844161_8129_4828_9e4c_2cc46c64adca.slice/crio-e435c5d1db3774032e555f023013d889400680a432e7cc1972b1c6c68e89f323 WatchSource:0}: Error finding container e435c5d1db3774032e555f023013d889400680a432e7cc1972b1c6c68e89f323: Status 404 returned error can't find the container with id e435c5d1db3774032e555f023013d889400680a432e7cc1972b1c6c68e89f323 Oct 31 21:25:19.060113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:19.060088 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:19.566154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:19.566061 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" event={"ID":"7c844161-8129-4828-9e4c-2cc46c64adca","Type":"ContainerStarted","Data":"1b514c103e87f552c7d17f198fb2a6c941f7b4fbdcb3fb2efb1ee025716242c4"} Oct 31 21:25:19.566154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:19.566118 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" event={"ID":"7c844161-8129-4828-9e4c-2cc46c64adca","Type":"ContainerStarted","Data":"e435c5d1db3774032e555f023013d889400680a432e7cc1972b1c6c68e89f323"} Oct 31 21:25:19.577056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:19.577028 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-2trls" Oct 31 21:25:19.608284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:19.608213 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" podStartSLOduration=1.402994869 podStartE2EDuration="1.608193296s" totalImagesPullingTime="205.198427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:25:19.6067541 +0000 UTC m=+566.299711494" watchObservedRunningTime="2026-10-31 21:25:19.608193296 +0000 UTC m=+566.301150689" Oct 31 21:25:19.756318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:19.756273 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:21.618933 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.618896 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4gck9"] Oct 31 21:25:21.622862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.622831 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.647731 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.647689 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 21:25:21.649993 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.649965 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4gck9"] Oct 31 21:25:21.745190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.745153 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.745395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.745197 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-home\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.745395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.745290 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.745472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.745396 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.745472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.745415 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6ncf\" (UniqueName: \"kubernetes.io/projected/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-kube-api-access-w6ncf\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.745555 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.745535 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.846226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.846191 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.846226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.846238 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.846495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.846258 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ncf\" (UniqueName: \"kubernetes.io/projected/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-kube-api-access-w6ncf\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.846495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.846311 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.846495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.846472 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.846655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.846540 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-home\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.846709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.846661 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.846709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.846695 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.846867 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.846846 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-home\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.848578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.848557 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.849019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.849000 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.867780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.867742 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6ncf\" (UniqueName: \"kubernetes.io/projected/a4686a76-fe03-456a-a669-6f8a1a6bbcf8-kube-api-access-w6ncf\") pod \"echo-simulator-kserve-64b6f8f96c-4gck9\" (UID: \"a4686a76-fe03-456a-a669-6f8a1a6bbcf8\") " pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:21.937219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:21.937133 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:22.241264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:22.241230 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-4gck9"] Oct 31 21:25:22.243498 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:25:22.243461 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4686a76_fe03_456a_a669_6f8a1a6bbcf8.slice/crio-ef7c2dfa3cdb0c5481bf44a01d3ec3474bf236824aafa976f98d736a64b8db25 WatchSource:0}: Error finding container ef7c2dfa3cdb0c5481bf44a01d3ec3474bf236824aafa976f98d736a64b8db25: Status 404 returned error can't find the container with id ef7c2dfa3cdb0c5481bf44a01d3ec3474bf236824aafa976f98d736a64b8db25 Oct 31 21:25:22.586791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:22.586714 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" event={"ID":"a4686a76-fe03-456a-a669-6f8a1a6bbcf8","Type":"ContainerStarted","Data":"ef7c2dfa3cdb0c5481bf44a01d3ec3474bf236824aafa976f98d736a64b8db25"} Oct 31 21:25:23.059368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.058446 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9"] Oct 31 21:25:23.079221 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.066314 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.089213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.088948 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 21:25:23.095349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.092181 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9"] Oct 31 21:25:23.160191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.160083 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82cpt\" (UniqueName: \"kubernetes.io/projected/a155f827-18b2-4296-bd24-4584460f57d4-kube-api-access-82cpt\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.160191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.160144 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.160472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.160212 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.160472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.160242 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.160472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.160269 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.160472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.160366 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a155f827-18b2-4296-bd24-4584460f57d4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.262025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.261240 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82cpt\" (UniqueName: \"kubernetes.io/projected/a155f827-18b2-4296-bd24-4584460f57d4-kube-api-access-82cpt\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.262025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.261296 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.262025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.261341 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.262025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.261373 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.262025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.261400 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.262025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.261430 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a155f827-18b2-4296-bd24-4584460f57d4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.262666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.262619 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.262747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.262706 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.262793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.262721 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.267376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.267311 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a155f827-18b2-4296-bd24-4584460f57d4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.273035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.273002 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a155f827-18b2-4296-bd24-4584460f57d4-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.292265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.292190 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82cpt\" (UniqueName: \"kubernetes.io/projected/a155f827-18b2-4296-bd24-4584460f57d4-kube-api-access-82cpt\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9\" (UID: \"a155f827-18b2-4296-bd24-4584460f57d4\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.389181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.389101 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:23.712972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:23.712931 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9"] Oct 31 21:25:23.714162 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:25:23.714117 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda155f827_18b2_4296_bd24_4584460f57d4.slice/crio-86cc690b33010c8acca868b8fb26245b6d8e7c18d4895232d13cfbc66d9a3fd8 WatchSource:0}: Error finding container 86cc690b33010c8acca868b8fb26245b6d8e7c18d4895232d13cfbc66d9a3fd8: Status 404 returned error can't find the container with id 86cc690b33010c8acca868b8fb26245b6d8e7c18d4895232d13cfbc66d9a3fd8 Oct 31 21:25:24.601604 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:24.601490 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" event={"ID":"a4686a76-fe03-456a-a669-6f8a1a6bbcf8","Type":"ContainerStarted","Data":"5549458daea41f3cd38b9b36cf5ee2848f8d7ee176b720d8cf61098f60b650e5"} Oct 31 21:25:24.603461 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:24.603421 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" event={"ID":"a155f827-18b2-4296-bd24-4584460f57d4","Type":"ContainerStarted","Data":"2ff429e893bfca10659fb62d843f49840507924ef3e5b37aa1d40d91e5e9f7e0"} Oct 31 21:25:24.603642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:24.603479 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" event={"ID":"a155f827-18b2-4296-bd24-4584460f57d4","Type":"ContainerStarted","Data":"86cc690b33010c8acca868b8fb26245b6d8e7c18d4895232d13cfbc66d9a3fd8"} Oct 31 21:25:24.642307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:24.642238 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" podStartSLOduration=1.697247695 podStartE2EDuration="3.642214185s" totalImagesPullingTime="1.94496649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:25:24.638716356 +0000 UTC m=+571.331673739" watchObservedRunningTime="2026-10-31 21:25:24.642214185 +0000 UTC m=+571.335171579" Oct 31 21:25:24.680038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:24.679966 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" podStartSLOduration=1.475985808 podStartE2EDuration="1.679945683s" totalImagesPullingTime="203.959875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:25:24.679035211 +0000 UTC m=+571.371992618" watchObservedRunningTime="2026-10-31 21:25:24.679945683 +0000 UTC m=+571.372903075" Oct 31 21:25:24.937792 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:24.937754 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:25.389457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:25.389425 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:28.756579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:28.756539 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:28.768855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:28.768813 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:29.643081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:29.643051 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf" Oct 31 21:25:31.937641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:31.937592 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:31.950172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:31.950140 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:32.537061 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:25:32.537022 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:25:32.659588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:32.659554 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-4gck9" Oct 31 21:25:33.389586 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:33.389548 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:33.402060 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:33.402025 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:33.670200 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:33.670115 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9" Oct 31 21:25:45.473620 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.473580 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8687975cdd-6l8tp"] Oct 31 21:25:45.474198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.473809 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-8687975cdd-6l8tp" podUID="f1a40dbf-9add-4726-b0e4-da271ad584f4" containerName="authorino" containerID="cri-o://bfc3ab26f927d8d12b086e2ab35e4473a89385143b2a5424e55930131720739e" gracePeriod=30 Oct 31 21:25:45.723161 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.723127 5188 generic.go:356] "Generic (PLEG): container finished" podID="f1a40dbf-9add-4726-b0e4-da271ad584f4" containerID="bfc3ab26f927d8d12b086e2ab35e4473a89385143b2a5424e55930131720739e" exitCode=0 Oct 31 21:25:45.723307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.723159 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8687975cdd-6l8tp" event={"ID":"f1a40dbf-9add-4726-b0e4-da271ad584f4","Type":"ContainerDied","Data":"bfc3ab26f927d8d12b086e2ab35e4473a89385143b2a5424e55930131720739e"} Oct 31 21:25:45.738114 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.738085 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:25:45.796552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.796491 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a40dbf-9add-4726-b0e4-da271ad584f4-kube-api-access-6qsp4\" (UniqueName: \"kubernetes.io/projected/f1a40dbf-9add-4726-b0e4-da271ad584f4-kube-api-access-6qsp4\") pod \"f1a40dbf-9add-4726-b0e4-da271ad584f4\" (UID: \"f1a40dbf-9add-4726-b0e4-da271ad584f4\") " Oct 31 21:25:45.798679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.798642 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a40dbf-9add-4726-b0e4-da271ad584f4-kube-api-access-6qsp4" pod "f1a40dbf-9add-4726-b0e4-da271ad584f4" (UID: "f1a40dbf-9add-4726-b0e4-da271ad584f4"). InnerVolumeSpecName "kube-api-access-6qsp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:25:45.897123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.897069 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a40dbf-9add-4726-b0e4-da271ad584f4-tls-cert\" (UniqueName: \"kubernetes.io/secret/f1a40dbf-9add-4726-b0e4-da271ad584f4-tls-cert\") pod \"f1a40dbf-9add-4726-b0e4-da271ad584f4\" (UID: \"f1a40dbf-9add-4726-b0e4-da271ad584f4\") " Oct 31 21:25:45.897345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.897311 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qsp4\" (UniqueName: \"kubernetes.io/projected/f1a40dbf-9add-4726-b0e4-da271ad584f4-kube-api-access-6qsp4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:25:45.908822 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.908787 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a40dbf-9add-4726-b0e4-da271ad584f4-tls-cert" pod "f1a40dbf-9add-4726-b0e4-da271ad584f4" (UID: "f1a40dbf-9add-4726-b0e4-da271ad584f4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:25:45.998613 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:45.998517 5188 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1a40dbf-9add-4726-b0e4-da271ad584f4-tls-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:25:46.731087 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:46.731056 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8687975cdd-6l8tp" Oct 31 21:25:46.731596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:46.731056 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8687975cdd-6l8tp" event={"ID":"f1a40dbf-9add-4726-b0e4-da271ad584f4","Type":"ContainerDied","Data":"8f60904e054e38795c486ab55c5b812a67a6d2d8d5b6375f75dc890f4e876f1c"} Oct 31 21:25:46.731596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:46.731218 5188 scope.go:122] "RemoveContainer" containerID="bfc3ab26f927d8d12b086e2ab35e4473a89385143b2a5424e55930131720739e" Oct 31 21:25:46.775315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:46.775232 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8687975cdd-6l8tp"] Oct 31 21:25:46.784989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:46.784959 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8687975cdd-6l8tp"] Oct 31 21:25:47.478781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:47.478743 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a40dbf-9add-4726-b0e4-da271ad584f4" path="/var/lib/kubelet/pods/f1a40dbf-9add-4726-b0e4-da271ad584f4/volumes" Oct 31 21:25:49.944604 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:49.944565 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx"] Oct 31 21:25:49.945046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:49.945027 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f1a40dbf-9add-4726-b0e4-da271ad584f4" containerName="authorino" Oct 31 21:25:49.945120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:49.945048 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a40dbf-9add-4726-b0e4-da271ad584f4" containerName="authorino" Oct 31 21:25:49.945212 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:49.945199 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a40dbf-9add-4726-b0e4-da271ad584f4" containerName="authorino" Oct 31 21:25:49.950455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:49.950425 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:49.973596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:49.973558 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dn2rx\"" Oct 31 21:25:49.973928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:49.973898 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 21:25:49.976551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:49.976523 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx"] Oct 31 21:25:50.036561 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.036520 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b314b11e-bba8-4bb2-978c-a53331086fa3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.036770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.036589 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.036770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.036699 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.036770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.036748 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.036938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.036855 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.036938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.036889 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.037013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.036962 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlh8\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-kube-api-access-hhlh8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.037013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.036985 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.037013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.037000 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.137816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.137772 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlh8\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-kube-api-access-hhlh8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.137816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.137816 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.137835 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.137865 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b314b11e-bba8-4bb2-978c-a53331086fa3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.137918 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.137944 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.137974 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.138084 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.138127 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.138501 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.138580 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138653 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.138610 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138746 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.138724 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.138831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.138812 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b314b11e-bba8-4bb2-978c-a53331086fa3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.140451 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.140427 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.140625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.140611 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.159282 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.159241 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.159464 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.159371 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlh8\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-kube-api-access-hhlh8\") pod \"maas-default-gateway-openshift-default-58847f9fc5-294zx\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.264487 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.264390 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:50.609575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.609532 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx"] Oct 31 21:25:50.611015 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:25:50.610988 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb314b11e_bba8_4bb2_978c_a53331086fa3.slice/crio-c054baaf548c8f84e84897f69f6fb61ca4b22660fb0697f0f2f4683ab385deec WatchSource:0}: Error finding container c054baaf548c8f84e84897f69f6fb61ca4b22660fb0697f0f2f4683ab385deec: Status 404 returned error can't find the container with id c054baaf548c8f84e84897f69f6fb61ca4b22660fb0697f0f2f4683ab385deec Oct 31 21:25:50.757944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:50.757909 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" event={"ID":"b314b11e-bba8-4bb2-978c-a53331086fa3","Type":"ContainerStarted","Data":"c054baaf548c8f84e84897f69f6fb61ca4b22660fb0697f0f2f4683ab385deec"} Oct 31 21:25:53.255067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:53.255022 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:25:53.255472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:53.255110 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:25:53.255472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:53.255145 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:25:53.600240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:53.600208 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:25:53.613625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:53.613599 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:25:53.778307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:53.778267 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" event={"ID":"b314b11e-bba8-4bb2-978c-a53331086fa3","Type":"ContainerStarted","Data":"5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3"} Oct 31 21:25:53.843410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:53.843293 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podStartSLOduration=2.202558681 podStartE2EDuration="4.843277514s" totalImagesPullingTime="2.640718833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:25:53.841275276 +0000 UTC m=+600.534232672" watchObservedRunningTime="2026-10-31 21:25:53.843277514 +0000 UTC m=+600.536234905" Oct 31 21:25:54.265478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:54.265440 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:25:54.267087 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:54.267057 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 21:25:54.267213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:54.267108 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 21:25:55.264773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:55.264732 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 21:25:55.264969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:55.264791 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 21:25:56.265646 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:56.265601 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 21:25:56.266046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:56.265664 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 21:25:57.265116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:57.265081 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 21:25:57.265295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:57.265140 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 21:25:58.265586 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:58.265547 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 21:25:58.265988 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:58.265603 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 21:25:59.265621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:59.265584 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 21:25:59.266024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:25:59.265653 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 21:26:00.265478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:00.265436 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 21:26:00.265697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:00.265512 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 21:26:01.448175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:01.448134 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:26:01.448656 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:01.448568 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:26:01.449537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:01.449515 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:26:02.536115 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:26:02.536077 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:26:04.996504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:04.996468 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j"] Oct 31 21:26:05.004056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.004033 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.015441 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.015415 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5"] Oct 31 21:26:05.019022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.019002 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.026534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.026513 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j"] Oct 31 21:26:05.047707 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.047654 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5"] Oct 31 21:26:05.099917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.099888 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.100100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.099927 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.100100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.099946 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.100100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100026 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.100100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100091 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.100284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100126 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.100284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100145 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.100284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100195 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.100284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100254 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.100423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100332 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7954fda0-ffa6-426f-b131-2b09ec6de759-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.100423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100374 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.100423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100399 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.100529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100433 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2hrj\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-kube-api-access-f2hrj\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.100529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100497 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvvv\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-kube-api-access-vnvvv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.100599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100528 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773ae86a-2248-47ba-a028-474ecbb56a97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.100599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100547 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.100659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100607 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773ae86a-2248-47ba-a028-474ecbb56a97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.100659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.100624 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201221 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201271 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201298 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201316 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201370 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201391 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201409 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7954fda0-ffa6-426f-b131-2b09ec6de759-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201439 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201463 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201517 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hrj\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-kube-api-access-f2hrj\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201538 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvvv\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-kube-api-access-vnvvv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201563 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773ae86a-2248-47ba-a028-474ecbb56a97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201592 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201642 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773ae86a-2248-47ba-a028-474ecbb56a97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201668 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201717 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201760 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201762 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201780 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201766 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.201971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.201766 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.202527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.202174 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.202527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.202314 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.202635 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.202600 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.202687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.202668 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7954fda0-ffa6-426f-b131-2b09ec6de759-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.202794 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.202773 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.202949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.202927 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.203281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.203257 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773ae86a-2248-47ba-a028-474ecbb56a97-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.203919 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.203892 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.204087 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.204066 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.204194 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.204178 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.204317 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.204296 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773ae86a-2248-47ba-a028-474ecbb56a97-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.222999 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.222970 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.223285 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.223261 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hrj\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-kube-api-access-f2hrj\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.223785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.223764 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-cts4j\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.223860 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.223791 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvvv\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-kube-api-access-vnvvv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nq6b5\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.317677 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.317577 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:05.334709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.334673 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:05.653372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.650633 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j"] Oct 31 21:26:05.657798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.653544 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:05.657798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.653680 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:05.657798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.655088 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5"] Oct 31 21:26:05.657798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.654934 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:05.657798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.655126 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:05.657798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.655204 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:05.657798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.655292 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:05.867779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.867101 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" event={"ID":"773ae86a-2248-47ba-a028-474ecbb56a97","Type":"ContainerStarted","Data":"2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e"} Oct 31 21:26:05.867779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.867176 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" event={"ID":"773ae86a-2248-47ba-a028-474ecbb56a97","Type":"ContainerStarted","Data":"9abed1318be9693f951cf5fd1f96123eb9da6336a66a0748e422b42eff7fc509"} Oct 31 21:26:05.869766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.869247 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" event={"ID":"7954fda0-ffa6-426f-b131-2b09ec6de759","Type":"ContainerStarted","Data":"cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe"} Oct 31 21:26:05.869766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.869299 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" event={"ID":"7954fda0-ffa6-426f-b131-2b09ec6de759","Type":"ContainerStarted","Data":"05f65c8e1483f82ad73039c951929b06de885b3063e78ce5a57688146f64a4b0"} Oct 31 21:26:05.910830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.907380 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podStartSLOduration=1.907361909 podStartE2EDuration="1.907361909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:05.906763534 +0000 UTC m=+612.599720930" watchObservedRunningTime="2026-10-31 21:26:05.907361909 +0000 UTC m=+612.600319303" Oct 31 21:26:05.948798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:05.948625 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podStartSLOduration=1.9486059820000001 podStartE2EDuration="1.948605982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:05.946717475 +0000 UTC m=+612.639674865" watchObservedRunningTime="2026-10-31 21:26:05.948605982 +0000 UTC m=+612.641563376" Oct 31 21:26:06.321098 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:06.321002 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:06.323376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:06.321476 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cts4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:26:06.323376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:06.321654 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 21:26:06.335943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:06.335906 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:06.337311 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:06.337283 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:26:06.337428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:06.337361 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 21:26:07.321000 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:07.320944 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cts4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:26:07.321388 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:07.321306 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 21:26:07.335339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:07.335290 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:26:07.335549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:07.335371 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 21:26:08.322350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:08.321728 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cts4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:26:08.322350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:08.321808 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 21:26:08.336098 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:08.336061 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:26:08.336269 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:08.336121 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 21:26:09.325964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:09.325923 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cts4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:26:09.326628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:09.326587 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 21:26:09.335564 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:09.335533 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:26:09.335760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:09.335740 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 21:26:10.320350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:10.319770 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cts4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:26:10.320350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:10.319843 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 21:26:10.335801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:10.335757 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:26:10.336182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:10.335823 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 21:26:11.321349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:11.320802 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cts4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:26:11.321349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:11.320880 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 21:26:11.335279 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:11.335238 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:26:11.335471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:11.335304 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 21:26:12.318220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:12.318182 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cts4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:26:12.318667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:12.318240 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Oct 31 21:26:12.335701 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:12.335665 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:26:12.335881 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:12.335733 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Oct 31 21:26:13.919967 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:13.919935 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:13.920490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:13.920429 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:13.921280 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:13.921260 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:26:13.958014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:13.957977 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:13.958462 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:13.958435 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:13.959512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:13.959494 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:26:15.455874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.455568 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr"] Oct 31 21:26:15.459761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.459730 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.473767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.473705 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-25447e22-openshift-default-dockercfg-fvpj7\"" Oct 31 21:26:15.486195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.486160 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr"] Oct 31 21:26:15.566231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.566189 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-socket\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.566451 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.566238 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istiod-ca-cert\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.567105 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.567045 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-envoy\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.567447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.567418 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-podinfo\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.567590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.567470 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-token\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.567964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.567943 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-certs\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.568154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.568135 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-credential-socket\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.568304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.568289 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-data\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.568511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.568479 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhw98\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-kube-api-access-dhw98\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.669704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.669659 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-socket\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.669903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.669712 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istiod-ca-cert\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.669903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.669769 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-envoy\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.669903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.669793 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-podinfo\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.669903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.669826 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-token\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.670116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.669911 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-certs\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.670116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.669968 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-credential-socket\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.670116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.669999 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-data\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.670116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.670024 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhw98\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-kube-api-access-dhw98\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.670340 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.670163 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-socket\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.670402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.670380 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-credential-socket\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.670540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.670513 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-data\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.670603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.670572 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istiod-ca-cert\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.670705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.670683 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-certs\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.672291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.672265 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-envoy\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.672535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.672512 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-podinfo\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.691553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.691513 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-token\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.691716 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.691662 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhw98\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-kube-api-access-dhw98\") pod \"e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.776643 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.776552 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:15.831957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.831914 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj"] Oct 31 21:26:15.850523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.850500 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:15.863627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.862741 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-353db9f5-gw-openshift-default-dockercfg-cns4x\"" Oct 31 21:26:15.864658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.864624 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj"] Oct 31 21:26:15.972757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.972716 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-data\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:15.972950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.972765 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-certs\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:15.972950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.972783 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-socket\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:15.972950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.972922 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fdf8\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-kube-api-access-5fdf8\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:15.972950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.972950 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-istio-token\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:15.973129 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.972982 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-envoy\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:15.973129 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.973073 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19e862e-98ff-497b-aafc-188726d8786f-istiod-ca-cert\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:15.973230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.973163 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19e862e-98ff-497b-aafc-188726d8786f-istio-podinfo\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:15.973230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:15.973205 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-credential-socket\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.074403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074256 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fdf8\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-kube-api-access-5fdf8\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.074403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074317 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-istio-token\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.074403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074384 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-envoy\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.074710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074415 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19e862e-98ff-497b-aafc-188726d8786f-istiod-ca-cert\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.074710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074442 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19e862e-98ff-497b-aafc-188726d8786f-istio-podinfo\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.074710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074467 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-credential-socket\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.074710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074507 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-data\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.074710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074551 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-certs\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.074710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074577 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-socket\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.075005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.074934 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-data\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.075064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.075016 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-credential-socket\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.075121 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.075086 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-certs\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.075217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.075192 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-socket\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.075217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.075198 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19e862e-98ff-497b-aafc-188726d8786f-istiod-ca-cert\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.076826 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.076802 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-envoy\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.076977 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.076959 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19e862e-98ff-497b-aafc-188726d8786f-istio-podinfo\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.095999 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.095963 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-istio-token\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.096152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.096087 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fdf8\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-kube-api-access-5fdf8\") pod \"e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.106506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.106480 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr"] Oct 31 21:26:16.107946 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:26:16.107915 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc98e0d_0445_4a04_b924_8da163c7f7e8.slice/crio-8495a84cbbb8330e497adf8ac8fba61234716af27c2ae238112f539cd4bb9bfb WatchSource:0}: Error finding container 8495a84cbbb8330e497adf8ac8fba61234716af27c2ae238112f539cd4bb9bfb: Status 404 returned error can't find the container with id 8495a84cbbb8330e497adf8ac8fba61234716af27c2ae238112f539cd4bb9bfb Oct 31 21:26:16.110053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.110024 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:16.110102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.110090 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:16.110138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.110117 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:16.185026 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.184993 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:16.503594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.503563 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj"] Oct 31 21:26:16.504806 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:26:16.504777 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19e862e_98ff_497b_aafc_188726d8786f.slice/crio-1f45f5d851b837a5b42167b2ad8c721ccda9b5bd70170c2647900b2d2e82094a WatchSource:0}: Error finding container 1f45f5d851b837a5b42167b2ad8c721ccda9b5bd70170c2647900b2d2e82094a: Status 404 returned error can't find the container with id 1f45f5d851b837a5b42167b2ad8c721ccda9b5bd70170c2647900b2d2e82094a Oct 31 21:26:16.507034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.506998 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:16.507108 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.507082 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:16.507152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.507128 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:16.992784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.992743 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" event={"ID":"c19e862e-98ff-497b-aafc-188726d8786f","Type":"ContainerStarted","Data":"a0c79e0d9a8ab607b4df3b21faa15c04659c0bb3058aa98f45dc2e79d379c207"} Oct 31 21:26:16.992952 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.992835 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" event={"ID":"c19e862e-98ff-497b-aafc-188726d8786f","Type":"ContainerStarted","Data":"1f45f5d851b837a5b42167b2ad8c721ccda9b5bd70170c2647900b2d2e82094a"} Oct 31 21:26:16.994468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.994441 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" event={"ID":"7bc98e0d-0445-4a04-b924-8da163c7f7e8","Type":"ContainerStarted","Data":"c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076"} Oct 31 21:26:16.994580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:16.994488 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" event={"ID":"7bc98e0d-0445-4a04-b924-8da163c7f7e8","Type":"ContainerStarted","Data":"8495a84cbbb8330e497adf8ac8fba61234716af27c2ae238112f539cd4bb9bfb"} Oct 31 21:26:17.041415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:17.041360 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" podStartSLOduration=2.041343184 podStartE2EDuration="2.041343184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:17.037814912 +0000 UTC m=+623.730772305" watchObservedRunningTime="2026-10-31 21:26:17.041343184 +0000 UTC m=+623.734300651" Oct 31 21:26:17.082037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:17.081782 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" podStartSLOduration=2.081763526 podStartE2EDuration="2.081763526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:17.081515771 +0000 UTC m=+623.774473166" watchObservedRunningTime="2026-10-31 21:26:17.081763526 +0000 UTC m=+623.774720919" Oct 31 21:26:17.185720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:17.185674 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:17.192232 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:17.192195 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:17.777261 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:17.777215 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:17.783186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:17.783154 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:18.000373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:18.000339 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:18.000574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:18.000399 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:18.001274 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:18.001250 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:18.001416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:18.001305 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:22.928717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:22.928676 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5"] Oct 31 21:26:22.933799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:22.933772 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:22.959589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:22.959547 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ac0736b6\"" Oct 31 21:26:22.959589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:22.959556 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ac0736b6-dockercfg-wjhnq\"" Oct 31 21:26:22.962671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:22.962641 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5"] Oct 31 21:26:22.991234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:22.991190 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5\" (UID: \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:22.991234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:22.991238 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msqp\" (UniqueName: \"kubernetes.io/projected/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-kube-api-access-9msqp\") pod \"payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5\" (UID: \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:23.005367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.005305 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm"] Oct 31 21:26:23.009191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.009162 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:23.031190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.031156 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm"] Oct 31 21:26:23.091931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.091883 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm\" (UID: \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:23.092119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.091951 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5\" (UID: \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:23.092119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.092029 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msqp\" (UniqueName: \"kubernetes.io/projected/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-kube-api-access-9msqp\") pod \"payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5\" (UID: \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:23.092261 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.092122 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvww8\" (UniqueName: \"kubernetes.io/projected/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-kube-api-access-bvww8\") pod \"payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm\" (UID: \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:23.092772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.092751 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5\" (UID: \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:23.113389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.113362 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msqp\" (UniqueName: \"kubernetes.io/projected/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-kube-api-access-9msqp\") pod \"payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5\" (UID: \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:23.193435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.193318 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm\" (UID: \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:23.193602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.193510 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvww8\" (UniqueName: \"kubernetes.io/projected/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-kube-api-access-bvww8\") pod \"payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm\" (UID: \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:23.194223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.194196 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm\" (UID: \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:23.229223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.217644 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvww8\" (UniqueName: \"kubernetes.io/projected/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-kube-api-access-bvww8\") pod \"payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm\" (UID: \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\") " pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:23.255353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.248893 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:23.326023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.325982 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:23.553164 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.553127 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5"] Oct 31 21:26:23.555064 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:26:23.555031 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b295b3a_fb65_41e3_bf2f_29d10796d5a2.slice/crio-a9693f2829067c92edd1047e483921c36f26ba6478de9585ef69b56d644ab4b4 WatchSource:0}: Error finding container a9693f2829067c92edd1047e483921c36f26ba6478de9585ef69b56d644ab4b4: Status 404 returned error can't find the container with id a9693f2829067c92edd1047e483921c36f26ba6478de9585ef69b56d644ab4b4 Oct 31 21:26:23.639670 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.639637 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm"] Oct 31 21:26:23.640307 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:26:23.640278 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d24e83_7da5_441c_b5ca_2dd28a344ff9.slice/crio-020bbcb2f3684ffee35f1d2e7deedcc8879858d2f9f7d35ed078c6340feb480f WatchSource:0}: Error finding container 020bbcb2f3684ffee35f1d2e7deedcc8879858d2f9f7d35ed078c6340feb480f: Status 404 returned error can't find the container with id 020bbcb2f3684ffee35f1d2e7deedcc8879858d2f9f7d35ed078c6340feb480f Oct 31 21:26:23.973792 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.973748 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q"] Oct 31 21:26:23.981720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.981691 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:23.995926 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:23.995892 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-7ef0e96a-openshift-default-dockercfg-dmg6b\"" Oct 31 21:26:24.011747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.011709 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q"] Oct 31 21:26:24.042435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.042404 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" event={"ID":"e6d24e83-7da5-441c-b5ca-2dd28a344ff9","Type":"ContainerStarted","Data":"ad92b13bb173ada944c4765be1fbc8a28aafdca93e7da930f595fca3201160c8"} Oct 31 21:26:24.042611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.042458 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" event={"ID":"e6d24e83-7da5-441c-b5ca-2dd28a344ff9","Type":"ContainerStarted","Data":"020bbcb2f3684ffee35f1d2e7deedcc8879858d2f9f7d35ed078c6340feb480f"} Oct 31 21:26:24.042611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.042513 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:24.044200 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.044175 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" event={"ID":"1b295b3a-fb65-41e3-bf2f-29d10796d5a2","Type":"ContainerStarted","Data":"c7ea03ff836185227bc15815bbc6778a641066289a0e50be0bb6d3a14c655c0c"} Oct 31 21:26:24.044304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.044213 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" event={"ID":"1b295b3a-fb65-41e3-bf2f-29d10796d5a2","Type":"ContainerStarted","Data":"a9693f2829067c92edd1047e483921c36f26ba6478de9585ef69b56d644ab4b4"} Oct 31 21:26:24.044304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.044234 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:24.086382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.079531 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" podStartSLOduration=2.079509206 podStartE2EDuration="2.079509206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:24.077899453 +0000 UTC m=+630.770856912" watchObservedRunningTime="2026-10-31 21:26:24.079509206 +0000 UTC m=+630.772466599" Oct 31 21:26:24.103779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.103461 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-credential-socket\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.103952 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.103897 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063ca5f5-b558-47b2-9960-bdebb49d83a3-istiod-ca-cert\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.107380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.104993 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-token\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.107380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.105112 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knpnm\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-kube-api-access-knpnm\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.107380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.105143 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-socket\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.107380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.105263 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-envoy\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.107380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.105451 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-data\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.107380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.105872 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-podinfo\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.107380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.105911 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-certs\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.117393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.113697 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" podStartSLOduration=2.113680261 podStartE2EDuration="2.113680261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:24.113115593 +0000 UTC m=+630.806072987" watchObservedRunningTime="2026-10-31 21:26:24.113680261 +0000 UTC m=+630.806637653" Oct 31 21:26:24.207045 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207009 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-data\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207045 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207049 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-podinfo\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207270 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207080 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-certs\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207270 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207139 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-credential-socket\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207270 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207187 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063ca5f5-b558-47b2-9960-bdebb49d83a3-istiod-ca-cert\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207270 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207259 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-token\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207305 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knpnm\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-kube-api-access-knpnm\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207378 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-socket\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207418 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-envoy\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207549 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-data\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207586 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-certs\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207648 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-credential-socket\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.207955 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.207931 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-socket\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.208149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.208128 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063ca5f5-b558-47b2-9960-bdebb49d83a3-istiod-ca-cert\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.209664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.209641 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-envoy\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.209763 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.209738 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-podinfo\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.230570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.230494 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-token\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.230570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.230497 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knpnm\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-kube-api-access-knpnm\") pod \"e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.299713 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.299675 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:24.619236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.619209 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q"] Oct 31 21:26:24.621530 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:26:24.621498 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063ca5f5_b558_47b2_9960_bdebb49d83a3.slice/crio-60dfefcad81d00200546d642b68d84a02fa1124dac8cc09e76fcf0b7b2962472 WatchSource:0}: Error finding container 60dfefcad81d00200546d642b68d84a02fa1124dac8cc09e76fcf0b7b2962472: Status 404 returned error can't find the container with id 60dfefcad81d00200546d642b68d84a02fa1124dac8cc09e76fcf0b7b2962472 Oct 31 21:26:24.623925 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.623890 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:24.624024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.623975 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:24.624024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:24.624015 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:25.051862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.051769 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" event={"ID":"063ca5f5-b558-47b2-9960-bdebb49d83a3","Type":"ContainerStarted","Data":"729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2"} Oct 31 21:26:25.051862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.051827 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" event={"ID":"063ca5f5-b558-47b2-9960-bdebb49d83a3","Type":"ContainerStarted","Data":"60dfefcad81d00200546d642b68d84a02fa1124dac8cc09e76fcf0b7b2962472"} Oct 31 21:26:25.112766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.112704 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" podStartSLOduration=2.11268831 podStartE2EDuration="2.11268831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:25.101205413 +0000 UTC m=+631.794162806" watchObservedRunningTime="2026-10-31 21:26:25.11268831 +0000 UTC m=+631.805645702" Oct 31 21:26:25.300566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.300526 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:25.306554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.306470 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:25.440033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.439973 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c"] Oct 31 21:26:25.444135 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.444112 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:25.457594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.457568 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9df039e4\"" Oct 31 21:26:25.458804 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.458785 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9df039e4-dockercfg-jlftl\"" Oct 31 21:26:25.479499 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.479462 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c"] Oct 31 21:26:25.524587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.524550 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bd90ec-1234-473d-8964-d82a3bcf83d6-plugins-config-volume\") pod \"payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c\" (UID: \"17bd90ec-1234-473d-8964-d82a3bcf83d6\") " pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:25.524797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.524753 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nkw\" (UniqueName: \"kubernetes.io/projected/17bd90ec-1234-473d-8964-d82a3bcf83d6-kube-api-access-n5nkw\") pod \"payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c\" (UID: \"17bd90ec-1234-473d-8964-d82a3bcf83d6\") " pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:25.625797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.625763 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bd90ec-1234-473d-8964-d82a3bcf83d6-plugins-config-volume\") pod \"payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c\" (UID: \"17bd90ec-1234-473d-8964-d82a3bcf83d6\") " pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:25.625967 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.625882 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nkw\" (UniqueName: \"kubernetes.io/projected/17bd90ec-1234-473d-8964-d82a3bcf83d6-kube-api-access-n5nkw\") pod \"payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c\" (UID: \"17bd90ec-1234-473d-8964-d82a3bcf83d6\") " pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:25.626600 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.626581 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bd90ec-1234-473d-8964-d82a3bcf83d6-plugins-config-volume\") pod \"payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c\" (UID: \"17bd90ec-1234-473d-8964-d82a3bcf83d6\") " pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:25.647494 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.647455 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nkw\" (UniqueName: \"kubernetes.io/projected/17bd90ec-1234-473d-8964-d82a3bcf83d6-kube-api-access-n5nkw\") pod \"payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c\" (UID: \"17bd90ec-1234-473d-8964-d82a3bcf83d6\") " pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:25.759036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:25.758998 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:26.058888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:26.058803 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:26.059881 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:26.059861 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:26.071192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:26.071165 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c"] Oct 31 21:26:26.072743 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:26:26.072708 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bd90ec_1234_473d_8964_d82a3bcf83d6.slice/crio-f6eddfa0e1662fa78b3d9ea3f379f38cbf8353b74c8e59fd42e7f0e6467cd8a0 WatchSource:0}: Error finding container f6eddfa0e1662fa78b3d9ea3f379f38cbf8353b74c8e59fd42e7f0e6467cd8a0: Status 404 returned error can't find the container with id f6eddfa0e1662fa78b3d9ea3f379f38cbf8353b74c8e59fd42e7f0e6467cd8a0 Oct 31 21:26:27.066883 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:27.066840 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" event={"ID":"17bd90ec-1234-473d-8964-d82a3bcf83d6","Type":"ContainerStarted","Data":"3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125"} Oct 31 21:26:27.067411 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:27.066909 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" event={"ID":"17bd90ec-1234-473d-8964-d82a3bcf83d6","Type":"ContainerStarted","Data":"f6eddfa0e1662fa78b3d9ea3f379f38cbf8353b74c8e59fd42e7f0e6467cd8a0"} Oct 31 21:26:27.067411 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:27.066940 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:27.107455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:27.107401 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" podStartSLOduration=2.107384751 podStartE2EDuration="2.107384751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:27.105283651 +0000 UTC m=+633.798241044" watchObservedRunningTime="2026-10-31 21:26:27.107384751 +0000 UTC m=+633.800342142" Oct 31 21:26:29.419635 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.419592 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr"] Oct 31 21:26:29.430715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.430682 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:29.442432 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.442387 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr"] Oct 31 21:26:29.447448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.447420 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7ef0e96a-dockercfg-9nlmm\"" Oct 31 21:26:29.448125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.448095 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7ef0e96a\"" Oct 31 21:26:29.509116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.509073 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc"] Oct 31 21:26:29.514954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.514926 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:29.536633 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.536597 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc"] Oct 31 21:26:29.620358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.619943 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de40230-111e-4929-b464-7a6a11a62310-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr\" (UID: \"9de40230-111e-4929-b464-7a6a11a62310\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:29.620358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.620046 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2484\" (UniqueName: \"kubernetes.io/projected/6132851e-aa58-4729-899c-9421e921879a-kube-api-access-n2484\") pod \"payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc\" (UID: \"6132851e-aa58-4729-899c-9421e921879a\") " pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:29.620358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.620178 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpc6\" (UniqueName: \"kubernetes.io/projected/9de40230-111e-4929-b464-7a6a11a62310-kube-api-access-hnpc6\") pod \"payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr\" (UID: \"9de40230-111e-4929-b464-7a6a11a62310\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:29.620358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.620236 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6132851e-aa58-4729-899c-9421e921879a-plugins-config-volume\") pod \"payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc\" (UID: \"6132851e-aa58-4729-899c-9421e921879a\") " pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:29.720886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.720787 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpc6\" (UniqueName: \"kubernetes.io/projected/9de40230-111e-4929-b464-7a6a11a62310-kube-api-access-hnpc6\") pod \"payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr\" (UID: \"9de40230-111e-4929-b464-7a6a11a62310\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:29.720886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.720854 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6132851e-aa58-4729-899c-9421e921879a-plugins-config-volume\") pod \"payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc\" (UID: \"6132851e-aa58-4729-899c-9421e921879a\") " pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:29.721146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.720906 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de40230-111e-4929-b464-7a6a11a62310-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr\" (UID: \"9de40230-111e-4929-b464-7a6a11a62310\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:29.721146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.720971 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2484\" (UniqueName: \"kubernetes.io/projected/6132851e-aa58-4729-899c-9421e921879a-kube-api-access-n2484\") pod \"payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc\" (UID: \"6132851e-aa58-4729-899c-9421e921879a\") " pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:29.721880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.721849 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6132851e-aa58-4729-899c-9421e921879a-plugins-config-volume\") pod \"payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc\" (UID: \"6132851e-aa58-4729-899c-9421e921879a\") " pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:29.721880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.721869 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de40230-111e-4929-b464-7a6a11a62310-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr\" (UID: \"9de40230-111e-4929-b464-7a6a11a62310\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:29.742142 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.742109 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2484\" (UniqueName: \"kubernetes.io/projected/6132851e-aa58-4729-899c-9421e921879a-kube-api-access-n2484\") pod \"payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc\" (UID: \"6132851e-aa58-4729-899c-9421e921879a\") " pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:29.742307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.742264 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpc6\" (UniqueName: \"kubernetes.io/projected/9de40230-111e-4929-b464-7a6a11a62310-kube-api-access-hnpc6\") pod \"payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr\" (UID: \"9de40230-111e-4929-b464-7a6a11a62310\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:29.747993 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.747959 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:29.828942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:29.828900 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:30.102295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:30.102258 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr"] Oct 31 21:26:30.113372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:30.111498 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" event={"ID":"9de40230-111e-4929-b464-7a6a11a62310","Type":"ContainerStarted","Data":"c39f4bd903c44ce73704036cb9cdc10487e67290d488af020067a97f746e0da1"} Oct 31 21:26:30.202688 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:30.195883 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc"] Oct 31 21:26:31.121485 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:31.121442 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" event={"ID":"9de40230-111e-4929-b464-7a6a11a62310","Type":"ContainerStarted","Data":"a5c952172b60b1c0f032a56fe13aa2abce1ee97fd2cb0aafec32efecdcb27aec"} Oct 31 21:26:31.121886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:31.121577 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:31.123614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:31.123584 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" event={"ID":"6132851e-aa58-4729-899c-9421e921879a","Type":"ContainerStarted","Data":"6ff6188b70c0a00b2bae8f12a20da62557e61dff6fc57e38819a756195c07356"} Oct 31 21:26:31.123770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:31.123629 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" event={"ID":"6132851e-aa58-4729-899c-9421e921879a","Type":"ContainerStarted","Data":"04079e0191fa357e763bfafb02912d18e94e939ee8bf4f53566f8d64779bdb2e"} Oct 31 21:26:31.123770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:31.123693 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:31.159197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:31.159139 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" podStartSLOduration=2.159123532 podStartE2EDuration="2.159123532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:31.156532495 +0000 UTC m=+637.849489888" watchObservedRunningTime="2026-10-31 21:26:31.159123532 +0000 UTC m=+637.852080923" Oct 31 21:26:31.198347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:31.198123 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" podStartSLOduration=2.198106768 podStartE2EDuration="2.198106768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:31.197861541 +0000 UTC m=+637.890818935" watchObservedRunningTime="2026-10-31 21:26:31.198106768 +0000 UTC m=+637.891064160" Oct 31 21:26:32.539735 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:26:32.539649 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:26:33.859517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.859470 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr"] Oct 31 21:26:33.863907 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.863884 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:33.867154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.867130 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:33.867297 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.867196 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmwtz\" (UniqueName: \"kubernetes.io/projected/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-kube-api-access-wmwtz\") pod \"maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:33.867384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.867296 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:33.904512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.904461 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr"] Oct 31 21:26:33.904693 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.904589 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:26:33.904765 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.904706 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:26:33.904818 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.904792 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:26:33.904868 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.904840 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:26:33.968173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.968133 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:33.968377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.968200 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:33.968377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.968255 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmwtz\" (UniqueName: \"kubernetes.io/projected/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-kube-api-access-wmwtz\") pod \"maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:33.968885 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.968858 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:33.970853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:33.970827 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:34.005279 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:34.005233 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmwtz\" (UniqueName: \"kubernetes.io/projected/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-kube-api-access-wmwtz\") pod \"maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:34.043561 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:34.043506 5188 patch_prober.go:29] interesting pod/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:26:34.043826 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:34.043596 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:26:34.216233 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:34.216189 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:34.565838 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:34.565786 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr"] Oct 31 21:26:34.568826 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:34.568788 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:26:35.056074 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:35.056032 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:26:35.057697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:35.057469 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:26:35.153846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:35.153803 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" event={"ID":"85811ba3-1441-4ae8-b6f4-d0911e57b0dd","Type":"ContainerStarted","Data":"fb0b2677b610f056194cdf5d44ce0a736af757b627e69120c8414c3f87b72543"} Oct 31 21:26:36.163001 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:36.162949 5188 generic.go:356] "Generic (PLEG): container finished" podID="c19e862e-98ff-497b-aafc-188726d8786f" containerID="a0c79e0d9a8ab607b4df3b21faa15c04659c0bb3058aa98f45dc2e79d379c207" exitCode=0 Oct 31 21:26:36.163540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:36.163032 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" event={"ID":"c19e862e-98ff-497b-aafc-188726d8786f","Type":"ContainerDied","Data":"a0c79e0d9a8ab607b4df3b21faa15c04659c0bb3058aa98f45dc2e79d379c207"} Oct 31 21:26:36.163598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:36.163574 5188 scope.go:122] "RemoveContainer" containerID="a0c79e0d9a8ab607b4df3b21faa15c04659c0bb3058aa98f45dc2e79d379c207" Oct 31 21:26:36.165159 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:36.165128 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:36.165301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:36.165221 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:36.165301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:36.165270 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:37.213710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:37.213558 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" event={"ID":"c19e862e-98ff-497b-aafc-188726d8786f","Type":"ContainerStarted","Data":"849115bfb9056085ad761d39583086b65b75c2675f83e5791658a6fb49f044c4"} Oct 31 21:26:38.103515 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:38.103483 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:38.193347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:38.186003 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:38.193347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:38.186131 5188 patch_prober.go:29] interesting pod/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:26:38.193347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:38.186188 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 21:26:39.187816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:39.187773 5188 patch_prober.go:29] interesting pod/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:26:39.188540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:39.188506 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 21:26:39.233734 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:39.233695 5188 generic.go:356] "Generic (PLEG): container finished" podID="85811ba3-1441-4ae8-b6f4-d0911e57b0dd" containerID="0dbef293373ec818c307b1e15af2938de8b2f69f8f1ea9b710420e4864382f3b" exitCode=0 Oct 31 21:26:39.233986 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:39.233959 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" event={"ID":"85811ba3-1441-4ae8-b6f4-d0911e57b0dd","Type":"ContainerDied","Data":"0dbef293373ec818c307b1e15af2938de8b2f69f8f1ea9b710420e4864382f3b"} Oct 31 21:26:40.186840 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.186801 5188 patch_prober.go:29] interesting pod/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:26:40.187130 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.187102 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 21:26:40.461941 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.461905 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:40.517964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.515291 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-xks-ca\") pod \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " Oct 31 21:26:40.517964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.515403 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-openshift-ca\") pod \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " Oct 31 21:26:40.517964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.515544 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-kube-api-access-wmwtz\" (UniqueName: \"kubernetes.io/projected/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-kube-api-access-wmwtz\") pod \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\" (UID: \"85811ba3-1441-4ae8-b6f4-d0911e57b0dd\") " Oct 31 21:26:40.517964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.516100 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-openshift-ca" pod "85811ba3-1441-4ae8-b6f4-d0911e57b0dd" (UID: "85811ba3-1441-4ae8-b6f4-d0911e57b0dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:26:40.517964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.516858 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:40.518782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.518740 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-xks-ca" pod "85811ba3-1441-4ae8-b6f4-d0911e57b0dd" (UID: "85811ba3-1441-4ae8-b6f4-d0911e57b0dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:26:40.531155 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.530129 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-kube-api-access-wmwtz" pod "85811ba3-1441-4ae8-b6f4-d0911e57b0dd" (UID: "85811ba3-1441-4ae8-b6f4-d0911e57b0dd"). InnerVolumeSpecName "kube-api-access-wmwtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:26:40.622254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.621659 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmwtz\" (UniqueName: \"kubernetes.io/projected/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-kube-api-access-wmwtz\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:40.622254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:40.621696 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85811ba3-1441-4ae8-b6f4-d0911e57b0dd-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:41.110812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.110769 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj"] Oct 31 21:26:41.112614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.111606 5188 cpu_manager.go:414] "containerMap: removing container" podUID="85811ba3-1441-4ae8-b6f4-d0911e57b0dd" containerName="revoke-keys" Oct 31 21:26:41.112614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.111642 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85811ba3-1441-4ae8-b6f4-d0911e57b0dd" containerName="revoke-keys" Oct 31 21:26:41.112614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.111929 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="85811ba3-1441-4ae8-b6f4-d0911e57b0dd" containerName="revoke-keys" Oct 31 21:26:41.116150 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.116111 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj"] Oct 31 21:26:41.116342 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.116231 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.195345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.194705 5188 patch_prober.go:29] interesting pod/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:26:41.195345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.194770 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 21:26:41.231309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.231274 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.231545 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.231362 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jrnd\" (UniqueName: \"kubernetes.io/projected/c6d03337-9069-4ac4-a587-8d5dce554678-kube-api-access-4jrnd\") pod \"maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.231618 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.231568 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.248658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.248618 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" Oct 31 21:26:41.248831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.248648 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr" event={"ID":"85811ba3-1441-4ae8-b6f4-d0911e57b0dd","Type":"ContainerDied","Data":"fb0b2677b610f056194cdf5d44ce0a736af757b627e69120c8414c3f87b72543"} Oct 31 21:26:41.248831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.248725 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb0b2677b610f056194cdf5d44ce0a736af757b627e69120c8414c3f87b72543" Oct 31 21:26:41.332675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.332630 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.332876 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.332723 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jrnd\" (UniqueName: \"kubernetes.io/projected/c6d03337-9069-4ac4-a587-8d5dce554678-kube-api-access-4jrnd\") pod \"maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.332876 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.332829 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.333894 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.333863 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.335753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.335720 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.354433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.354394 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jrnd\" (UniqueName: \"kubernetes.io/projected/c6d03337-9069-4ac4-a587-8d5dce554678-kube-api-access-4jrnd\") pod \"maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.437122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.437025 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:41.807546 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:41.807502 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj"] Oct 31 21:26:41.809958 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:26:41.809920 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d03337_9069_4ac4_a587_8d5dce554678.slice/crio-a4e489892232da82a8710020003ce793be109215c31f1cd805df7da794338f83 WatchSource:0}: Error finding container a4e489892232da82a8710020003ce793be109215c31f1cd805df7da794338f83: Status 404 returned error can't find the container with id a4e489892232da82a8710020003ce793be109215c31f1cd805df7da794338f83 Oct 31 21:26:42.134208 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.134166 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:26:42.134477 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.134366 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:26:42.211381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.210410 5188 patch_prober.go:29] interesting pod/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:26:42.211381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.210477 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 31 21:26:42.265827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.265784 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" event={"ID":"c6d03337-9069-4ac4-a587-8d5dce554678","Type":"ContainerStarted","Data":"0cf06ee2f76b24f2ee5c0dbee41da1c1e36933e0d13634bbe9518996dc30b53b"} Oct 31 21:26:42.265997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.265859 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" event={"ID":"c6d03337-9069-4ac4-a587-8d5dce554678","Type":"ContainerStarted","Data":"a4e489892232da82a8710020003ce793be109215c31f1cd805df7da794338f83"} Oct 31 21:26:42.269240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.269210 5188 generic.go:356] "Generic (PLEG): container finished" podID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerID="c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076" exitCode=0 Oct 31 21:26:42.269378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.269346 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" event={"ID":"7bc98e0d-0445-4a04-b924-8da163c7f7e8","Type":"ContainerDied","Data":"c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076"} Oct 31 21:26:42.269833 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.269813 5188 scope.go:122] "RemoveContainer" containerID="c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076" Oct 31 21:26:42.277366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.271567 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:42.277366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.271660 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:42.277366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:42.271708 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:43.200402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:43.199647 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:43.216349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:43.201054 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:43.216349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:43.203534 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:26:43.310349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:43.303420 5188 generic.go:356] "Generic (PLEG): container finished" podID="c6d03337-9069-4ac4-a587-8d5dce554678" containerID="0cf06ee2f76b24f2ee5c0dbee41da1c1e36933e0d13634bbe9518996dc30b53b" exitCode=0 Oct 31 21:26:43.310349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:43.303589 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" event={"ID":"c6d03337-9069-4ac4-a587-8d5dce554678","Type":"ContainerDied","Data":"0cf06ee2f76b24f2ee5c0dbee41da1c1e36933e0d13634bbe9518996dc30b53b"} Oct 31 21:26:43.328355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:43.315973 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" event={"ID":"7bc98e0d-0445-4a04-b924-8da163c7f7e8","Type":"ContainerStarted","Data":"762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb"} Oct 31 21:26:43.787933 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:43.783239 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:43.792346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:43.789448 5188 patch_prober.go:29] interesting pod/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:26:43.792346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:43.789524 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 21:26:44.780721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:44.777760 5188 patch_prober.go:29] interesting pod/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:26:44.780721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:44.777817 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 21:26:44.853210 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:44.851418 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx"] Oct 31 21:26:44.952584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:44.952542 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx"] Oct 31 21:26:44.952762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:44.952673 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:44.957939 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:44.957915 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:45.070095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.067118 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-openshift-ca\") pod \"c6d03337-9069-4ac4-a587-8d5dce554678\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " Oct 31 21:26:45.070095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.067189 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d03337-9069-4ac4-a587-8d5dce554678-kube-api-access-4jrnd\" (UniqueName: \"kubernetes.io/projected/c6d03337-9069-4ac4-a587-8d5dce554678-kube-api-access-4jrnd\") pod \"c6d03337-9069-4ac4-a587-8d5dce554678\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " Oct 31 21:26:45.070095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.067273 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-xks-ca\") pod \"c6d03337-9069-4ac4-a587-8d5dce554678\" (UID: \"c6d03337-9069-4ac4-a587-8d5dce554678\") " Oct 31 21:26:45.070095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.067975 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.070095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.068043 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pncm4\" (UniqueName: \"kubernetes.io/projected/8b62b327-13de-463a-bc74-b284e01fb2ad-kube-api-access-pncm4\") pod \"maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.070095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.068153 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.070095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.068622 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-openshift-ca" pod "c6d03337-9069-4ac4-a587-8d5dce554678" (UID: "c6d03337-9069-4ac4-a587-8d5dce554678"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:26:45.108627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.108496 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d03337-9069-4ac4-a587-8d5dce554678-kube-api-access-4jrnd" pod "c6d03337-9069-4ac4-a587-8d5dce554678" (UID: "c6d03337-9069-4ac4-a587-8d5dce554678"). InnerVolumeSpecName "kube-api-access-4jrnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:26:45.108627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.108592 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-xks-ca" pod "c6d03337-9069-4ac4-a587-8d5dce554678" (UID: "c6d03337-9069-4ac4-a587-8d5dce554678"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:26:45.169268 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.169223 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.169495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.169314 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.169495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.169376 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pncm4\" (UniqueName: \"kubernetes.io/projected/8b62b327-13de-463a-bc74-b284e01fb2ad-kube-api-access-pncm4\") pod \"maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.169495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.169435 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:45.169495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.169446 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jrnd\" (UniqueName: \"kubernetes.io/projected/c6d03337-9069-4ac4-a587-8d5dce554678-kube-api-access-4jrnd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:45.169495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.169455 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d03337-9069-4ac4-a587-8d5dce554678-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:45.170593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.170565 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.172842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.172817 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.196171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.196101 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pncm4\" (UniqueName: \"kubernetes.io/projected/8b62b327-13de-463a-bc74-b284e01fb2ad-kube-api-access-pncm4\") pod \"maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.267516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.267478 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:45.305833 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.305781 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm"] Oct 31 21:26:45.306869 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.306837 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c6d03337-9069-4ac4-a587-8d5dce554678" containerName="revoke-keys" Oct 31 21:26:45.307040 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.307025 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d03337-9069-4ac4-a587-8d5dce554678" containerName="revoke-keys" Oct 31 21:26:45.307307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.307289 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d03337-9069-4ac4-a587-8d5dce554678" containerName="revoke-keys" Oct 31 21:26:45.310107 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.310086 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.342476 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.342382 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm"] Oct 31 21:26:45.414369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.390745 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7blpq\" (UniqueName: \"kubernetes.io/projected/5b9704cd-481b-42e3-b41f-b5f56a111e4f-kube-api-access-7blpq\") pod \"maas-api-revoke-keys-e2e-ait-25447e22-6wsbm\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.414369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.390822 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-25447e22-6wsbm\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.414369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.390873 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-25447e22-6wsbm\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.470660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.467854 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" Oct 31 21:26:45.547458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.544015 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7blpq\" (UniqueName: \"kubernetes.io/projected/5b9704cd-481b-42e3-b41f-b5f56a111e4f-kube-api-access-7blpq\") pod \"maas-api-revoke-keys-e2e-ait-25447e22-6wsbm\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.547458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.544084 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-25447e22-6wsbm\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.547458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.544134 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-25447e22-6wsbm\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.547458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.545023 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-25447e22-6wsbm\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.558959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.558923 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-25447e22-6wsbm\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.562930 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.562716 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj" event={"ID":"c6d03337-9069-4ac4-a587-8d5dce554678","Type":"ContainerDied","Data":"a4e489892232da82a8710020003ce793be109215c31f1cd805df7da794338f83"} Oct 31 21:26:45.562930 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.562781 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e489892232da82a8710020003ce793be109215c31f1cd805df7da794338f83" Oct 31 21:26:45.567606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.567579 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7blpq\" (UniqueName: \"kubernetes.io/projected/5b9704cd-481b-42e3-b41f-b5f56a111e4f-kube-api-access-7blpq\") pod \"maas-api-revoke-keys-e2e-ait-25447e22-6wsbm\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.746316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.746076 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:45.802369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.779810 5188 patch_prober.go:29] interesting pod/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:26:45.802369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.779892 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 21:26:45.946905 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:26:45.946849 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b62b327_13de_463a_bc74_b284e01fb2ad.slice/crio-f2b435c743ef977788a859ee192f803f7881b499bef9877436a24e3ef602fbc4 WatchSource:0}: Error finding container f2b435c743ef977788a859ee192f803f7881b499bef9877436a24e3ef602fbc4: Status 404 returned error can't find the container with id f2b435c743ef977788a859ee192f803f7881b499bef9877436a24e3ef602fbc4 Oct 31 21:26:45.948056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:45.948027 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx"] Oct 31 21:26:46.140031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:46.139995 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm"] Oct 31 21:26:46.485766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:46.485667 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" event={"ID":"8b62b327-13de-463a-bc74-b284e01fb2ad","Type":"ContainerStarted","Data":"d031ab1db19664299c3a50b54ce3a9fda1855f328f06f32dc840fe4b3e1ab812"} Oct 31 21:26:46.486018 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:46.486001 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" event={"ID":"8b62b327-13de-463a-bc74-b284e01fb2ad","Type":"ContainerStarted","Data":"f2b435c743ef977788a859ee192f803f7881b499bef9877436a24e3ef602fbc4"} Oct 31 21:26:46.504678 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:46.504638 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" event={"ID":"5b9704cd-481b-42e3-b41f-b5f56a111e4f","Type":"ContainerStarted","Data":"a3870bfc41d774d57a6fb489d03507d012673eb3d96f94e94a43b71c39d1d753"} Oct 31 21:26:46.784366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:46.780076 5188 patch_prober.go:29] interesting pod/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:26:46.784366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:46.780141 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 21:26:47.530937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:47.530863 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" event={"ID":"5b9704cd-481b-42e3-b41f-b5f56a111e4f","Type":"ContainerStarted","Data":"7ba789117b76d530614543820382103b4157db80f509a4db5563225e071409cb"} Oct 31 21:26:47.533483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:47.533459 5188 generic.go:356] "Generic (PLEG): container finished" podID="8b62b327-13de-463a-bc74-b284e01fb2ad" containerID="d031ab1db19664299c3a50b54ce3a9fda1855f328f06f32dc840fe4b3e1ab812" exitCode=0 Oct 31 21:26:47.533568 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:47.533497 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" event={"ID":"8b62b327-13de-463a-bc74-b284e01fb2ad","Type":"ContainerDied","Data":"d031ab1db19664299c3a50b54ce3a9fda1855f328f06f32dc840fe4b3e1ab812"} Oct 31 21:26:47.599348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:47.598049 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" podStartSLOduration=3.598027445 podStartE2EDuration="3.598027445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:46.531632203 +0000 UTC m=+653.224589596" watchObservedRunningTime="2026-10-31 21:26:47.598027445 +0000 UTC m=+654.290984837" Oct 31 21:26:47.780807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:47.780452 5188 patch_prober.go:29] interesting pod/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:26:47.780807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:47.780520 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Oct 31 21:26:48.000429 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.000376 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:48.545832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.545782 5188 generic.go:356] "Generic (PLEG): container finished" podID="5b9704cd-481b-42e3-b41f-b5f56a111e4f" containerID="7ba789117b76d530614543820382103b4157db80f509a4db5563225e071409cb" exitCode=0 Oct 31 21:26:48.546394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.546366 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" event={"ID":"5b9704cd-481b-42e3-b41f-b5f56a111e4f","Type":"ContainerDied","Data":"7ba789117b76d530614543820382103b4157db80f509a4db5563225e071409cb"} Oct 31 21:26:48.743690 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.743660 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:48.786809 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.784818 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:48.787647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.787624 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:26:48.796008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.795943 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:48.902100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.902061 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-xks-ca\") pod \"8b62b327-13de-463a-bc74-b284e01fb2ad\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " Oct 31 21:26:48.902306 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.902294 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b9704cd-481b-42e3-b41f-b5f56a111e4f-kube-api-access-7blpq\" (UniqueName: \"kubernetes.io/projected/5b9704cd-481b-42e3-b41f-b5f56a111e4f-kube-api-access-7blpq\") pod \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " Oct 31 21:26:48.902702 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.902677 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-openshift-ca\") pod \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " Oct 31 21:26:48.902837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.902720 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-xks-ca\") pod \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\" (UID: \"5b9704cd-481b-42e3-b41f-b5f56a111e4f\") " Oct 31 21:26:48.902837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.902775 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-openshift-ca\") pod \"8b62b327-13de-463a-bc74-b284e01fb2ad\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " Oct 31 21:26:48.902837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.902832 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b62b327-13de-463a-bc74-b284e01fb2ad-kube-api-access-pncm4\" (UniqueName: \"kubernetes.io/projected/8b62b327-13de-463a-bc74-b284e01fb2ad-kube-api-access-pncm4\") pod \"8b62b327-13de-463a-bc74-b284e01fb2ad\" (UID: \"8b62b327-13de-463a-bc74-b284e01fb2ad\") " Oct 31 21:26:48.904242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.903779 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-openshift-ca" pod "8b62b327-13de-463a-bc74-b284e01fb2ad" (UID: "8b62b327-13de-463a-bc74-b284e01fb2ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:26:48.904242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.904010 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-openshift-ca" pod "5b9704cd-481b-42e3-b41f-b5f56a111e4f" (UID: "5b9704cd-481b-42e3-b41f-b5f56a111e4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:26:48.908622 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.908582 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b62b327-13de-463a-bc74-b284e01fb2ad-kube-api-access-pncm4" pod "8b62b327-13de-463a-bc74-b284e01fb2ad" (UID: "8b62b327-13de-463a-bc74-b284e01fb2ad"). InnerVolumeSpecName "kube-api-access-pncm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:26:48.909125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.909089 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-xks-ca" pod "5b9704cd-481b-42e3-b41f-b5f56a111e4f" (UID: "5b9704cd-481b-42e3-b41f-b5f56a111e4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:26:48.911580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.911543 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-xks-ca" pod "8b62b327-13de-463a-bc74-b284e01fb2ad" (UID: "8b62b327-13de-463a-bc74-b284e01fb2ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:26:48.913781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:48.913746 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9704cd-481b-42e3-b41f-b5f56a111e4f-kube-api-access-7blpq" pod "5b9704cd-481b-42e3-b41f-b5f56a111e4f" (UID: "5b9704cd-481b-42e3-b41f-b5f56a111e4f"). InnerVolumeSpecName "kube-api-access-7blpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:26:49.005048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.005010 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7blpq\" (UniqueName: \"kubernetes.io/projected/5b9704cd-481b-42e3-b41f-b5f56a111e4f-kube-api-access-7blpq\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:49.005048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.005050 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:49.005248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.005067 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b9704cd-481b-42e3-b41f-b5f56a111e4f-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:49.005248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.005083 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:49.005248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.005097 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pncm4\" (UniqueName: \"kubernetes.io/projected/8b62b327-13de-463a-bc74-b284e01fb2ad-kube-api-access-pncm4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:49.005248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.005110 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b62b327-13de-463a-bc74-b284e01fb2ad-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:49.556042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.556006 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" Oct 31 21:26:49.556042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.556030 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx" event={"ID":"8b62b327-13de-463a-bc74-b284e01fb2ad","Type":"ContainerDied","Data":"f2b435c743ef977788a859ee192f803f7881b499bef9877436a24e3ef602fbc4"} Oct 31 21:26:49.556512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.556089 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b435c743ef977788a859ee192f803f7881b499bef9877436a24e3ef602fbc4" Oct 31 21:26:49.557728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.557699 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" Oct 31 21:26:49.557866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.557736 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm" event={"ID":"5b9704cd-481b-42e3-b41f-b5f56a111e4f","Type":"ContainerDied","Data":"a3870bfc41d774d57a6fb489d03507d012673eb3d96f94e94a43b71c39d1d753"} Oct 31 21:26:49.557866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:49.557782 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3870bfc41d774d57a6fb489d03507d012673eb3d96f94e94a43b71c39d1d753" Oct 31 21:26:51.571600 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.571513 5188 generic.go:356] "Generic (PLEG): container finished" podID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerID="729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2" exitCode=0 Oct 31 21:26:51.572086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.571596 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" event={"ID":"063ca5f5-b558-47b2-9960-bdebb49d83a3","Type":"ContainerDied","Data":"729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2"} Oct 31 21:26:51.572086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.572080 5188 scope.go:122] "RemoveContainer" containerID="729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2" Oct 31 21:26:51.573474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.573450 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:51.573538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.573517 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:51.573576 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.573547 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:26:51.635421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.635393 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c"] Oct 31 21:26:51.987479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.987374 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk"] Oct 31 21:26:51.989111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.989079 5188 cpu_manager.go:414] "containerMap: removing container" podUID="8b62b327-13de-463a-bc74-b284e01fb2ad" containerName="revoke-keys" Oct 31 21:26:51.989111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.989109 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b62b327-13de-463a-bc74-b284e01fb2ad" containerName="revoke-keys" Oct 31 21:26:51.989337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.989163 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5b9704cd-481b-42e3-b41f-b5f56a111e4f" containerName="revoke-keys" Oct 31 21:26:51.989337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.989172 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b9704cd-481b-42e3-b41f-b5f56a111e4f" containerName="revoke-keys" Oct 31 21:26:51.989868 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.989839 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b62b327-13de-463a-bc74-b284e01fb2ad" containerName="revoke-keys" Oct 31 21:26:51.989960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.989879 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b9704cd-481b-42e3-b41f-b5f56a111e4f" containerName="revoke-keys" Oct 31 21:26:51.992864 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:51.992836 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.003861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.003785 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk"] Oct 31 21:26:52.005964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.005935 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:26:52.006100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.006023 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:26:52.006508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.006158 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:26:52.006508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.006230 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:26:52.139995 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.139954 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4007579e-9720-43df-a819-406502242771-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk\" (UID: \"4007579e-9720-43df-a819-406502242771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.140185 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.140123 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4p7\" (UniqueName: \"kubernetes.io/projected/4007579e-9720-43df-a819-406502242771-kube-api-access-wb4p7\") pod \"maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk\" (UID: \"4007579e-9720-43df-a819-406502242771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.140185 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.140166 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4007579e-9720-43df-a819-406502242771-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk\" (UID: \"4007579e-9720-43df-a819-406502242771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.240911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.240800 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4p7\" (UniqueName: \"kubernetes.io/projected/4007579e-9720-43df-a819-406502242771-kube-api-access-wb4p7\") pod \"maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk\" (UID: \"4007579e-9720-43df-a819-406502242771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.240911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.240856 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4007579e-9720-43df-a819-406502242771-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk\" (UID: \"4007579e-9720-43df-a819-406502242771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.241171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.240948 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4007579e-9720-43df-a819-406502242771-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk\" (UID: \"4007579e-9720-43df-a819-406502242771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.241838 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.241810 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4007579e-9720-43df-a819-406502242771-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk\" (UID: \"4007579e-9720-43df-a819-406502242771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.243435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.243411 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4007579e-9720-43df-a819-406502242771-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk\" (UID: \"4007579e-9720-43df-a819-406502242771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.262491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.262458 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4p7\" (UniqueName: \"kubernetes.io/projected/4007579e-9720-43df-a819-406502242771-kube-api-access-wb4p7\") pod \"maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk\" (UID: \"4007579e-9720-43df-a819-406502242771\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.316801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.316501 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:52.604626 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.604539 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" event={"ID":"063ca5f5-b558-47b2-9960-bdebb49d83a3","Type":"ContainerStarted","Data":"c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d"} Oct 31 21:26:52.644991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:52.644963 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk"] Oct 31 21:26:52.651460 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:26:52.651426 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4007579e_9720_43df_a819_406502242771.slice/crio-c08e1a53ed2367bf1a62878b2980d71a46a67ff1e59929808a2908311fa951e5 WatchSource:0}: Error finding container c08e1a53ed2367bf1a62878b2980d71a46a67ff1e59929808a2908311fa951e5: Status 404 returned error can't find the container with id c08e1a53ed2367bf1a62878b2980d71a46a67ff1e59929808a2908311fa951e5 Oct 31 21:26:53.300149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:53.300035 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:53.301407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:53.301384 5188 patch_prober.go:29] interesting pod/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:26:53.301490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:53.301435 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 21:26:53.611777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:53.611743 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" event={"ID":"4007579e-9720-43df-a819-406502242771","Type":"ContainerStarted","Data":"30e246d69625b34da3f21e16781dfe6bb6d5d9f775fa13121d32e6e7ff3d2dae"} Oct 31 21:26:53.612156 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:53.611801 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" event={"ID":"4007579e-9720-43df-a819-406502242771","Type":"ContainerStarted","Data":"c08e1a53ed2367bf1a62878b2980d71a46a67ff1e59929808a2908311fa951e5"} Oct 31 21:26:53.650459 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:53.650400 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" podStartSLOduration=2.650376614 podStartE2EDuration="2.650376614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:53.648385687 +0000 UTC m=+660.341343078" watchObservedRunningTime="2026-10-31 21:26:53.650376614 +0000 UTC m=+660.343334007" Oct 31 21:26:54.301140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:54.301100 5188 patch_prober.go:29] interesting pod/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:26:54.301339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:54.301160 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 21:26:54.619596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:54.619560 5188 generic.go:356] "Generic (PLEG): container finished" podID="4007579e-9720-43df-a819-406502242771" containerID="30e246d69625b34da3f21e16781dfe6bb6d5d9f775fa13121d32e6e7ff3d2dae" exitCode=0 Oct 31 21:26:54.620010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:54.619652 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" event={"ID":"4007579e-9720-43df-a819-406502242771","Type":"ContainerDied","Data":"30e246d69625b34da3f21e16781dfe6bb6d5d9f775fa13121d32e6e7ff3d2dae"} Oct 31 21:26:55.302833 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.302786 5188 patch_prober.go:29] interesting pod/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:26:55.303049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.302871 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 21:26:55.340209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.340173 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr"] Oct 31 21:26:55.349883 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.349845 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-353db9f5-m48cr"] Oct 31 21:26:55.352780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.352748 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm"] Oct 31 21:26:55.403484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.403448 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5"] Oct 31 21:26:55.515880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.515839 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85811ba3-1441-4ae8-b6f4-d0911e57b0dd" path="/var/lib/kubelet/pods/85811ba3-1441-4ae8-b6f4-d0911e57b0dd/volumes" Oct 31 21:26:55.533377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.532064 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj"] Oct 31 21:26:55.533377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.532877 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" containerID="cri-o://849115bfb9056085ad761d39583086b65b75c2675f83e5791658a6fb49f044c4" gracePeriod=30 Oct 31 21:26:55.861922 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:55.861895 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:56.001953 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.001865 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj"] Oct 31 21:26:56.005707 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.005668 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9df039e4-6pwkj"] Oct 31 21:26:56.014365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.012427 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4007579e-9720-43df-a819-406502242771-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4007579e-9720-43df-a819-406502242771-maas-api-openshift-ca\") pod \"4007579e-9720-43df-a819-406502242771\" (UID: \"4007579e-9720-43df-a819-406502242771\") " Oct 31 21:26:56.014365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.012529 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4007579e-9720-43df-a819-406502242771-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4007579e-9720-43df-a819-406502242771-maas-api-xks-ca\") pod \"4007579e-9720-43df-a819-406502242771\" (UID: \"4007579e-9720-43df-a819-406502242771\") " Oct 31 21:26:56.014365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.012563 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4007579e-9720-43df-a819-406502242771-kube-api-access-wb4p7\" (UniqueName: \"kubernetes.io/projected/4007579e-9720-43df-a819-406502242771-kube-api-access-wb4p7\") pod \"4007579e-9720-43df-a819-406502242771\" (UID: \"4007579e-9720-43df-a819-406502242771\") " Oct 31 21:26:56.014365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.013591 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4007579e-9720-43df-a819-406502242771-maas-api-openshift-ca" pod "4007579e-9720-43df-a819-406502242771" (UID: "4007579e-9720-43df-a819-406502242771"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:26:56.027654 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.024930 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4007579e-9720-43df-a819-406502242771-kube-api-access-wb4p7" pod "4007579e-9720-43df-a819-406502242771" (UID: "4007579e-9720-43df-a819-406502242771"). InnerVolumeSpecName "kube-api-access-wb4p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:26:56.038210 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.028536 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4007579e-9720-43df-a819-406502242771-maas-api-xks-ca" pod "4007579e-9720-43df-a819-406502242771" (UID: "4007579e-9720-43df-a819-406502242771"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:26:56.059293 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.059253 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:56.113502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.113468 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4007579e-9720-43df-a819-406502242771-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:56.113502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.113498 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4p7\" (UniqueName: \"kubernetes.io/projected/4007579e-9720-43df-a819-406502242771-kube-api-access-wb4p7\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:56.113766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.113507 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4007579e-9720-43df-a819-406502242771-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:56.307170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.306861 5188 patch_prober.go:29] interesting pod/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:26:56.307170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.306938 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 21:26:56.636144 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.636095 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" podUID="17bd90ec-1234-473d-8964-d82a3bcf83d6" containerName="payload-processing" containerID="cri-o://3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125" gracePeriod=25 Oct 31 21:26:56.649802 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.649769 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" Oct 31 21:26:56.649942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.649869 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk" event={"ID":"4007579e-9720-43df-a819-406502242771","Type":"ContainerDied","Data":"c08e1a53ed2367bf1a62878b2980d71a46a67ff1e59929808a2908311fa951e5"} Oct 31 21:26:56.649942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:56.649921 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c08e1a53ed2367bf1a62878b2980d71a46a67ff1e59929808a2908311fa951e5" Oct 31 21:26:57.015897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.015607 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:57.127580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.127531 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17bd90ec-1234-473d-8964-d82a3bcf83d6-kube-api-access-n5nkw\" (UniqueName: \"kubernetes.io/projected/17bd90ec-1234-473d-8964-d82a3bcf83d6-kube-api-access-n5nkw\") pod \"17bd90ec-1234-473d-8964-d82a3bcf83d6\" (UID: \"17bd90ec-1234-473d-8964-d82a3bcf83d6\") " Oct 31 21:26:57.127742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.127695 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17bd90ec-1234-473d-8964-d82a3bcf83d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bd90ec-1234-473d-8964-d82a3bcf83d6-plugins-config-volume\") pod \"17bd90ec-1234-473d-8964-d82a3bcf83d6\" (UID: \"17bd90ec-1234-473d-8964-d82a3bcf83d6\") " Oct 31 21:26:57.128144 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.128122 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17bd90ec-1234-473d-8964-d82a3bcf83d6-plugins-config-volume" pod "17bd90ec-1234-473d-8964-d82a3bcf83d6" (UID: "17bd90ec-1234-473d-8964-d82a3bcf83d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:26:57.132398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.132358 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bd90ec-1234-473d-8964-d82a3bcf83d6-kube-api-access-n5nkw" pod "17bd90ec-1234-473d-8964-d82a3bcf83d6" (UID: "17bd90ec-1234-473d-8964-d82a3bcf83d6"). InnerVolumeSpecName "kube-api-access-n5nkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:26:57.229600 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.229511 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17bd90ec-1234-473d-8964-d82a3bcf83d6-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:57.229600 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.229541 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5nkw\" (UniqueName: \"kubernetes.io/projected/17bd90ec-1234-473d-8964-d82a3bcf83d6-kube-api-access-n5nkw\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:26:57.300559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.300516 5188 patch_prober.go:29] interesting pod/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:26:57.300770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.300585 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Oct 31 21:26:57.475103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.475057 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d03337-9069-4ac4-a587-8d5dce554678" path="/var/lib/kubelet/pods/c6d03337-9069-4ac4-a587-8d5dce554678/volumes" Oct 31 21:26:57.658351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.658304 5188 generic.go:356] "Generic (PLEG): container finished" podID="17bd90ec-1234-473d-8964-d82a3bcf83d6" containerID="3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125" exitCode=0 Oct 31 21:26:57.658544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.658434 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" Oct 31 21:26:57.658544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.658450 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" event={"ID":"17bd90ec-1234-473d-8964-d82a3bcf83d6","Type":"ContainerDied","Data":"3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125"} Oct 31 21:26:57.658544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.658497 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c" event={"ID":"17bd90ec-1234-473d-8964-d82a3bcf83d6","Type":"ContainerDied","Data":"f6eddfa0e1662fa78b3d9ea3f379f38cbf8353b74c8e59fd42e7f0e6467cd8a0"} Oct 31 21:26:57.658544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.658517 5188 scope.go:122] "RemoveContainer" containerID="3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125" Oct 31 21:26:57.669227 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.669201 5188 scope.go:122] "RemoveContainer" containerID="3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125" Oct 31 21:26:57.669659 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:26:57.669637 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125\": container with ID starting with 3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125 not found: ID does not exist" containerID="3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125" Oct 31 21:26:57.669730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.669669 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125"} err="failed to get container status \"3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125\": rpc error: code = NotFound desc = could not find container \"3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125\": container with ID starting with 3216aa424c9293b437b465de0c212a457973c72cf88c83604b82e82fe514f125 not found: ID does not exist" Oct 31 21:26:57.704894 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.704860 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c"] Oct 31 21:26:57.708943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:57.708914 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9df039e4-84bfcd89ff-bkr7c"] Oct 31 21:26:58.204121 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:58.204084 5188 patch_prober.go:29] interesting pod/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:26:58.204512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:58.204142 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:26:58.305864 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:58.305826 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:58.306932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:58.306905 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:26:59.473389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:26:59.473348 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17bd90ec-1234-473d-8964-d82a3bcf83d6" path="/var/lib/kubelet/pods/17bd90ec-1234-473d-8964-d82a3bcf83d6/volumes" Oct 31 21:27:00.353524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.353480 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" podUID="e6d24e83-7da5-441c-b5ca-2dd28a344ff9" containerName="payload-processing" containerID="cri-o://ad92b13bb173ada944c4765be1fbc8a28aafdca93e7da930f595fca3201160c8" gracePeriod=25 Oct 31 21:27:00.359487 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.359456 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm"] Oct 31 21:27:00.368430 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.368392 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-25447e22-6wsbm"] Oct 31 21:27:00.404528 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.404477 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" podUID="1b295b3a-fb65-41e3-bf2f-29d10796d5a2" containerName="payload-pre-processing" containerID="cri-o://c7ea03ff836185227bc15815bbc6778a641066289a0e50be0bb6d3a14c655c0c" gracePeriod=25 Oct 31 21:27:00.683604 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.683572 5188 generic.go:356] "Generic (PLEG): container finished" podID="c19e862e-98ff-497b-aafc-188726d8786f" containerID="849115bfb9056085ad761d39583086b65b75c2675f83e5791658a6fb49f044c4" exitCode=0 Oct 31 21:27:00.684060 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.683761 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" event={"ID":"c19e862e-98ff-497b-aafc-188726d8786f","Type":"ContainerDied","Data":"849115bfb9056085ad761d39583086b65b75c2675f83e5791658a6fb49f044c4"} Oct 31 21:27:00.684060 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.683859 5188 scope.go:122] "RemoveContainer" containerID="a0c79e0d9a8ab607b4df3b21faa15c04659c0bb3058aa98f45dc2e79d379c207" Oct 31 21:27:00.687003 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.686980 5188 generic.go:356] "Generic (PLEG): container finished" podID="e6d24e83-7da5-441c-b5ca-2dd28a344ff9" containerID="ad92b13bb173ada944c4765be1fbc8a28aafdca93e7da930f595fca3201160c8" exitCode=0 Oct 31 21:27:00.687139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.687114 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" event={"ID":"e6d24e83-7da5-441c-b5ca-2dd28a344ff9","Type":"ContainerDied","Data":"ad92b13bb173ada944c4765be1fbc8a28aafdca93e7da930f595fca3201160c8"} Oct 31 21:27:00.690012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.689983 5188 generic.go:356] "Generic (PLEG): container finished" podID="1b295b3a-fb65-41e3-bf2f-29d10796d5a2" containerID="c7ea03ff836185227bc15815bbc6778a641066289a0e50be0bb6d3a14c655c0c" exitCode=0 Oct 31 21:27:00.690154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.690039 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" event={"ID":"1b295b3a-fb65-41e3-bf2f-29d10796d5a2","Type":"ContainerDied","Data":"c7ea03ff836185227bc15815bbc6778a641066289a0e50be0bb6d3a14c655c0c"} Oct 31 21:27:00.715855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.715831 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:27:00.724281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.724216 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:27:00.769633 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.769595 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-plugins-config-volume\") pod \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\" (UID: \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\") " Oct 31 21:27:00.769807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.769665 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-kube-api-access-9msqp\" (UniqueName: \"kubernetes.io/projected/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-kube-api-access-9msqp\") pod \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\" (UID: \"1b295b3a-fb65-41e3-bf2f-29d10796d5a2\") " Oct 31 21:27:00.769807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.769719 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-kube-api-access-bvww8\" (UniqueName: \"kubernetes.io/projected/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-kube-api-access-bvww8\") pod \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\" (UID: \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\") " Oct 31 21:27:00.769928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.769838 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-plugins-config-volume\") pod \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\" (UID: \"e6d24e83-7da5-441c-b5ca-2dd28a344ff9\") " Oct 31 21:27:00.770008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.769983 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-plugins-config-volume" pod "1b295b3a-fb65-41e3-bf2f-29d10796d5a2" (UID: "1b295b3a-fb65-41e3-bf2f-29d10796d5a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:27:00.770362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.770333 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.770509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.770484 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-plugins-config-volume" pod "e6d24e83-7da5-441c-b5ca-2dd28a344ff9" (UID: "e6d24e83-7da5-441c-b5ca-2dd28a344ff9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:27:00.771993 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.771964 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-kube-api-access-9msqp" pod "1b295b3a-fb65-41e3-bf2f-29d10796d5a2" (UID: "1b295b3a-fb65-41e3-bf2f-29d10796d5a2"). InnerVolumeSpecName "kube-api-access-9msqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:00.772254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.772225 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-kube-api-access-bvww8" pod "e6d24e83-7da5-441c-b5ca-2dd28a344ff9" (UID: "e6d24e83-7da5-441c-b5ca-2dd28a344ff9"). InnerVolumeSpecName "kube-api-access-bvww8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:00.822988 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.822953 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx"] Oct 31 21:27:00.825764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.825733 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ac0736b6-g4pjx"] Oct 31 21:27:00.829865 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.829842 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:27:00.870722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.870649 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c19e862e-98ff-497b-aafc-188726d8786f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19e862e-98ff-497b-aafc-188726d8786f-istio-podinfo\") pod \"c19e862e-98ff-497b-aafc-188726d8786f\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " Oct 31 21:27:00.870910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.870772 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-data\") pod \"c19e862e-98ff-497b-aafc-188726d8786f\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " Oct 31 21:27:00.870910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.870800 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-certs\") pod \"c19e862e-98ff-497b-aafc-188726d8786f\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " Oct 31 21:27:00.870910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.870828 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c19e862e-98ff-497b-aafc-188726d8786f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19e862e-98ff-497b-aafc-188726d8786f-istiod-ca-cert\") pod \"c19e862e-98ff-497b-aafc-188726d8786f\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " Oct 31 21:27:00.870910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.870848 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-kube-api-access-5fdf8\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-kube-api-access-5fdf8\") pod \"c19e862e-98ff-497b-aafc-188726d8786f\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " Oct 31 21:27:00.870910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.870902 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-istio-token\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-istio-token\") pod \"c19e862e-98ff-497b-aafc-188726d8786f\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " Oct 31 21:27:00.871114 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.870952 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-envoy\") pod \"c19e862e-98ff-497b-aafc-188726d8786f\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " Oct 31 21:27:00.871114 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.870981 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-credential-socket\") pod \"c19e862e-98ff-497b-aafc-188726d8786f\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " Oct 31 21:27:00.871114 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.871059 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-socket\") pod \"c19e862e-98ff-497b-aafc-188726d8786f\" (UID: \"c19e862e-98ff-497b-aafc-188726d8786f\") " Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.871513 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msqp\" (UniqueName: \"kubernetes.io/projected/1b295b3a-fb65-41e3-bf2f-29d10796d5a2-kube-api-access-9msqp\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.871545 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvww8\" (UniqueName: \"kubernetes.io/projected/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-kube-api-access-bvww8\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.871564 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6d24e83-7da5-441c-b5ca-2dd28a344ff9-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.871786 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-socket" pod "c19e862e-98ff-497b-aafc-188726d8786f" (UID: "c19e862e-98ff-497b-aafc-188726d8786f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.871983 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-credential-socket" pod "c19e862e-98ff-497b-aafc-188726d8786f" (UID: "c19e862e-98ff-497b-aafc-188726d8786f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.872249 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19e862e-98ff-497b-aafc-188726d8786f-istiod-ca-cert" pod "c19e862e-98ff-497b-aafc-188726d8786f" (UID: "c19e862e-98ff-497b-aafc-188726d8786f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.872267 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-certs" pod "c19e862e-98ff-497b-aafc-188726d8786f" (UID: "c19e862e-98ff-497b-aafc-188726d8786f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.872351 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-data" pod "c19e862e-98ff-497b-aafc-188726d8786f" (UID: "c19e862e-98ff-497b-aafc-188726d8786f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.873345 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c19e862e-98ff-497b-aafc-188726d8786f-istio-podinfo" pod "c19e862e-98ff-497b-aafc-188726d8786f" (UID: "c19e862e-98ff-497b-aafc-188726d8786f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.875000 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-istio-token" pod "c19e862e-98ff-497b-aafc-188726d8786f" (UID: "c19e862e-98ff-497b-aafc-188726d8786f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:00.875152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.875056 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-kube-api-access-5fdf8" pod "c19e862e-98ff-497b-aafc-188726d8786f" (UID: "c19e862e-98ff-497b-aafc-188726d8786f"). InnerVolumeSpecName "kube-api-access-5fdf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:00.875871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.875848 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-envoy" pod "c19e862e-98ff-497b-aafc-188726d8786f" (UID: "c19e862e-98ff-497b-aafc-188726d8786f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:00.976437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.976027 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.976437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.976067 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19e862e-98ff-497b-aafc-188726d8786f-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.976437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.976081 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.976437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.976094 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.976437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.976107 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19e862e-98ff-497b-aafc-188726d8786f-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.976437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.976120 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fdf8\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-kube-api-access-5fdf8\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.976437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.976135 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19e862e-98ff-497b-aafc-188726d8786f-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.976437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.976150 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.976437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:00.976162 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19e862e-98ff-497b-aafc-188726d8786f-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:01.472549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.472513 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b9704cd-481b-42e3-b41f-b5f56a111e4f" path="/var/lib/kubelet/pods/5b9704cd-481b-42e3-b41f-b5f56a111e4f/volumes" Oct 31 21:27:01.472927 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.472913 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b62b327-13de-463a-bc74-b284e01fb2ad" path="/var/lib/kubelet/pods/8b62b327-13de-463a-bc74-b284e01fb2ad/volumes" Oct 31 21:27:01.699617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.699510 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" Oct 31 21:27:01.699617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.699539 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm" event={"ID":"e6d24e83-7da5-441c-b5ca-2dd28a344ff9","Type":"ContainerDied","Data":"020bbcb2f3684ffee35f1d2e7deedcc8879858d2f9f7d35ed078c6340feb480f"} Oct 31 21:27:01.699617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.699612 5188 scope.go:122] "RemoveContainer" containerID="ad92b13bb173ada944c4765be1fbc8a28aafdca93e7da930f595fca3201160c8" Oct 31 21:27:01.701441 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.701419 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" Oct 31 21:27:01.702085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.701959 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5" event={"ID":"1b295b3a-fb65-41e3-bf2f-29d10796d5a2","Type":"ContainerDied","Data":"a9693f2829067c92edd1047e483921c36f26ba6478de9585ef69b56d644ab4b4"} Oct 31 21:27:01.704169 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.704132 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" event={"ID":"c19e862e-98ff-497b-aafc-188726d8786f","Type":"ContainerDied","Data":"1f45f5d851b837a5b42167b2ad8c721ccda9b5bd70170c2647900b2d2e82094a"} Oct 31 21:27:01.704420 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.704398 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj" Oct 31 21:27:01.710874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.710846 5188 scope.go:122] "RemoveContainer" containerID="c7ea03ff836185227bc15815bbc6778a641066289a0e50be0bb6d3a14c655c0c" Oct 31 21:27:01.720424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.720386 5188 scope.go:122] "RemoveContainer" containerID="849115bfb9056085ad761d39583086b65b75c2675f83e5791658a6fb49f044c4" Oct 31 21:27:01.745987 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.745955 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj"] Oct 31 21:27:01.756031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.755996 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-353db9f5-gw-openshift-default-7d885d947c-k9fxj"] Oct 31 21:27:01.780589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.780551 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm"] Oct 31 21:27:01.788421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.788386 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac0736b6-dc9cbf5c9-xsqhm"] Oct 31 21:27:01.817771 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.817738 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5"] Oct 31 21:27:01.821694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.821660 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac0736b6-54b457b99-m5xt5"] Oct 31 21:27:01.896052 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.896019 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr"] Oct 31 21:27:01.896277 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:01.896248 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" containerID="cri-o://762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb" gracePeriod=30 Oct 31 21:27:02.269345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269287 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96"] Oct 31 21:27:02.269806 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269790 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" Oct 31 21:27:02.269848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269808 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" Oct 31 21:27:02.269848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269819 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e6d24e83-7da5-441c-b5ca-2dd28a344ff9" containerName="payload-processing" Oct 31 21:27:02.269848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269824 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d24e83-7da5-441c-b5ca-2dd28a344ff9" containerName="payload-processing" Oct 31 21:27:02.269848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269844 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" Oct 31 21:27:02.269966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269849 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" Oct 31 21:27:02.269966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269857 5188 cpu_manager.go:414] "containerMap: removing container" podUID="4007579e-9720-43df-a819-406502242771" containerName="revoke-keys" Oct 31 21:27:02.269966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269862 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4007579e-9720-43df-a819-406502242771" containerName="revoke-keys" Oct 31 21:27:02.269966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269876 5188 cpu_manager.go:414] "containerMap: removing container" podUID="1b295b3a-fb65-41e3-bf2f-29d10796d5a2" containerName="payload-pre-processing" Oct 31 21:27:02.269966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269881 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b295b3a-fb65-41e3-bf2f-29d10796d5a2" containerName="payload-pre-processing" Oct 31 21:27:02.269966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269898 5188 cpu_manager.go:414] "containerMap: removing container" podUID="17bd90ec-1234-473d-8964-d82a3bcf83d6" containerName="payload-processing" Oct 31 21:27:02.269966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269902 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17bd90ec-1234-473d-8964-d82a3bcf83d6" containerName="payload-processing" Oct 31 21:27:02.269966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269963 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" Oct 31 21:27:02.270191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269970 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="4007579e-9720-43df-a819-406502242771" containerName="revoke-keys" Oct 31 21:27:02.270191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269978 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b295b3a-fb65-41e3-bf2f-29d10796d5a2" containerName="payload-pre-processing" Oct 31 21:27:02.270191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269986 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="17bd90ec-1234-473d-8964-d82a3bcf83d6" containerName="payload-processing" Oct 31 21:27:02.270191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269993 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19e862e-98ff-497b-aafc-188726d8786f" containerName="istio-proxy" Oct 31 21:27:02.270191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.269999 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d24e83-7da5-441c-b5ca-2dd28a344ff9" containerName="payload-processing" Oct 31 21:27:02.275051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.275023 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.289503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.289466 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.289669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.289600 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2829db12-be6a-46ba-a4d9-58158af0f3b4-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.289739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.289663 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.289739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.289707 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdvfg\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-kube-api-access-hdvfg\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.289843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.289770 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.289843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.289828 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.289939 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.289921 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.289975 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.289964 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.290038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.290023 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.293851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.293828 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c16684-openshift-default-dockercfg-t7nbj\"" Oct 31 21:27:02.297988 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.297960 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96"] Oct 31 21:27:02.391653 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.391613 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2829db12-be6a-46ba-a4d9-58158af0f3b4-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.391974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.391906 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.392173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.392146 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdvfg\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-kube-api-access-hdvfg\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.392275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.392186 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.392275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.392243 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.392433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.392337 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.392433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.392398 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.392530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.392509 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.392578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.392540 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.393073 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.393049 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.393858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.393836 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2829db12-be6a-46ba-a4d9-58158af0f3b4-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.394101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.394083 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.394690 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.394669 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.395619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.395572 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.402541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.402511 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.406610 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.406582 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.437573 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.437535 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.437737 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.437684 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdvfg\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-kube-api-access-hdvfg\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.537192 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:27:02.537092 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:27:02.570870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.570834 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc"] Oct 31 21:27:02.591506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.591467 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:02.605063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.605029 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr"] Oct 31 21:27:02.932873 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.932837 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96"] Oct 31 21:27:02.933641 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:27:02.933609 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2829db12_be6a_46ba_a4d9_58158af0f3b4.slice/crio-f36311135e288c3def72d9c5333272a415abe27860033c65016151a1075de209 WatchSource:0}: Error finding container f36311135e288c3def72d9c5333272a415abe27860033c65016151a1075de209: Status 404 returned error can't find the container with id f36311135e288c3def72d9c5333272a415abe27860033c65016151a1075de209 Oct 31 21:27:02.935947 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.935911 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:02.936021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.935980 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:02.936021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:02.936007 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:03.472936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:03.472899 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b295b3a-fb65-41e3-bf2f-29d10796d5a2" path="/var/lib/kubelet/pods/1b295b3a-fb65-41e3-bf2f-29d10796d5a2/volumes" Oct 31 21:27:03.473353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:03.473336 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c19e862e-98ff-497b-aafc-188726d8786f" path="/var/lib/kubelet/pods/c19e862e-98ff-497b-aafc-188726d8786f/volumes" Oct 31 21:27:03.473677 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:03.473664 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d24e83-7da5-441c-b5ca-2dd28a344ff9" path="/var/lib/kubelet/pods/e6d24e83-7da5-441c-b5ca-2dd28a344ff9/volumes" Oct 31 21:27:03.746841 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:03.746752 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" event={"ID":"2829db12-be6a-46ba-a4d9-58158af0f3b4","Type":"ContainerStarted","Data":"2205f7189c45a4ae05943c1a68ec455137ef1cbfcc0f9eaa38e43cbb9e58760c"} Oct 31 21:27:03.746841 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:03.746817 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" event={"ID":"2829db12-be6a-46ba-a4d9-58158af0f3b4","Type":"ContainerStarted","Data":"f36311135e288c3def72d9c5333272a415abe27860033c65016151a1075de209"} Oct 31 21:27:03.789058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:03.789017 5188 patch_prober.go:29] interesting pod/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:27:03.789205 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:03.789083 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:27:03.797455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:03.797392 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" podStartSLOduration=1.797373129 podStartE2EDuration="1.797373129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:03.795189811 +0000 UTC m=+670.488147217" watchObservedRunningTime="2026-10-31 21:27:03.797373129 +0000 UTC m=+670.490330523" Oct 31 21:27:04.592068 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:04.592034 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:04.597444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:04.597413 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:04.753572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:04.753540 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:04.754468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:04.754446 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:05.068994 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.068953 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b"] Oct 31 21:27:05.074720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.074691 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:05.093364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.093304 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4b5989\"" Oct 31 21:27:05.093730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.093705 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4b5989-dockercfg-7rdqm\"" Oct 31 21:27:05.094281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.094253 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b"] Oct 31 21:27:05.125366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.125309 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqhb\" (UniqueName: \"kubernetes.io/projected/4433702b-b648-49e8-9aa5-12bcdc71b275-kube-api-access-cbqhb\") pod \"payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b\" (UID: \"4433702b-b648-49e8-9aa5-12bcdc71b275\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:05.125555 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.125538 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4433702b-b648-49e8-9aa5-12bcdc71b275-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b\" (UID: \"4433702b-b648-49e8-9aa5-12bcdc71b275\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:05.226231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.226191 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4433702b-b648-49e8-9aa5-12bcdc71b275-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b\" (UID: \"4433702b-b648-49e8-9aa5-12bcdc71b275\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:05.226431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.226259 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqhb\" (UniqueName: \"kubernetes.io/projected/4433702b-b648-49e8-9aa5-12bcdc71b275-kube-api-access-cbqhb\") pod \"payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b\" (UID: \"4433702b-b648-49e8-9aa5-12bcdc71b275\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:05.227037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.227015 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4433702b-b648-49e8-9aa5-12bcdc71b275-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b\" (UID: \"4433702b-b648-49e8-9aa5-12bcdc71b275\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:05.253361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.248566 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqhb\" (UniqueName: \"kubernetes.io/projected/4433702b-b648-49e8-9aa5-12bcdc71b275-kube-api-access-cbqhb\") pod \"payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b\" (UID: \"4433702b-b648-49e8-9aa5-12bcdc71b275\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:05.392689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.392642 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:05.699131 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.699087 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b"] Oct 31 21:27:05.700587 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:27:05.700556 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4433702b_b648_49e8_9aa5_12bcdc71b275.slice/crio-565d309e985ce3bd851166f324fb5c6010eebca9aa1ef384b9b0f8666366a1af WatchSource:0}: Error finding container 565d309e985ce3bd851166f324fb5c6010eebca9aa1ef384b9b0f8666366a1af: Status 404 returned error can't find the container with id 565d309e985ce3bd851166f324fb5c6010eebca9aa1ef384b9b0f8666366a1af Oct 31 21:27:05.761173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:05.761142 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" event={"ID":"4433702b-b648-49e8-9aa5-12bcdc71b275","Type":"ContainerStarted","Data":"565d309e985ce3bd851166f324fb5c6010eebca9aa1ef384b9b0f8666366a1af"} Oct 31 21:27:06.223774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:06.223742 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk"] Oct 31 21:27:06.234923 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:06.234890 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7ef0e96a-jsxtk"] Oct 31 21:27:06.388861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:06.388740 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q"] Oct 31 21:27:06.389559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:06.389522 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" containerID="cri-o://c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d" gracePeriod=30 Oct 31 21:27:06.769652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:06.769605 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" event={"ID":"4433702b-b648-49e8-9aa5-12bcdc71b275","Type":"ContainerStarted","Data":"0bb1d391c9765c248113dd339f208296a9a7987c99edc7f6f6439816c1172415"} Oct 31 21:27:06.770054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:06.769692 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:06.819715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:06.819656 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" podStartSLOduration=1.819642097 podStartE2EDuration="1.819642097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:06.819318444 +0000 UTC m=+673.512275837" watchObservedRunningTime="2026-10-31 21:27:06.819642097 +0000 UTC m=+673.512599487" Oct 31 21:27:07.168015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.167988 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:27:07.252423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252388 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-socket\") pod \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " Oct 31 21:27:07.252590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252452 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-certs\") pod \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " Oct 31 21:27:07.252590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252513 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-credential-socket\") pod \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " Oct 31 21:27:07.252590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252547 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-podinfo\") pod \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " Oct 31 21:27:07.252762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252609 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istiod-ca-cert\") pod \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " Oct 31 21:27:07.252762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252649 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-kube-api-access-dhw98\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-kube-api-access-dhw98\") pod \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " Oct 31 21:27:07.252762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252705 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-data\") pod \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " Oct 31 21:27:07.252762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252744 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-envoy\") pod \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " Oct 31 21:27:07.252762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252696 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-socket" pod "7bc98e0d-0445-4a04-b924-8da163c7f7e8" (UID: "7bc98e0d-0445-4a04-b924-8da163c7f7e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:07.253011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252793 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-certs" pod "7bc98e0d-0445-4a04-b924-8da163c7f7e8" (UID: "7bc98e0d-0445-4a04-b924-8da163c7f7e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:07.253011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252878 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-token\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-token\") pod \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\" (UID: \"7bc98e0d-0445-4a04-b924-8da163c7f7e8\") " Oct 31 21:27:07.253011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.252891 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-credential-socket" pod "7bc98e0d-0445-4a04-b924-8da163c7f7e8" (UID: "7bc98e0d-0445-4a04-b924-8da163c7f7e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:07.253154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.253018 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-data" pod "7bc98e0d-0445-4a04-b924-8da163c7f7e8" (UID: "7bc98e0d-0445-4a04-b924-8da163c7f7e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:07.253154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.253056 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istiod-ca-cert" pod "7bc98e0d-0445-4a04-b924-8da163c7f7e8" (UID: "7bc98e0d-0445-4a04-b924-8da163c7f7e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:27:07.253352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.253302 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:07.253352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.253343 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:07.253484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.253357 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:07.253484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.253372 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:07.253484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.253385 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:07.255055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.255007 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-token" pod "7bc98e0d-0445-4a04-b924-8da163c7f7e8" (UID: "7bc98e0d-0445-4a04-b924-8da163c7f7e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:07.255055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.255007 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-envoy" pod "7bc98e0d-0445-4a04-b924-8da163c7f7e8" (UID: "7bc98e0d-0445-4a04-b924-8da163c7f7e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:07.255055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.255038 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-kube-api-access-dhw98" pod "7bc98e0d-0445-4a04-b924-8da163c7f7e8" (UID: "7bc98e0d-0445-4a04-b924-8da163c7f7e8"). InnerVolumeSpecName "kube-api-access-dhw98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:07.255201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.255092 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-podinfo" pod "7bc98e0d-0445-4a04-b924-8da163c7f7e8" (UID: "7bc98e0d-0445-4a04-b924-8da163c7f7e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:27:07.354397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.354281 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhw98\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-kube-api-access-dhw98\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:07.354397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.354316 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:07.354397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.354356 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:07.354397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.354365 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bc98e0d-0445-4a04-b924-8da163c7f7e8-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:07.474078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.474043 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4007579e-9720-43df-a819-406502242771" path="/var/lib/kubelet/pods/4007579e-9720-43df-a819-406502242771/volumes" Oct 31 21:27:07.571501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.571461 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" podUID="6132851e-aa58-4729-899c-9421e921879a" containerName="payload-processing" containerID="cri-o://6ff6188b70c0a00b2bae8f12a20da62557e61dff6fc57e38819a756195c07356" gracePeriod=25 Oct 31 21:27:07.606173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.606070 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" podUID="9de40230-111e-4929-b464-7a6a11a62310" containerName="payload-pre-processing" containerID="cri-o://a5c952172b60b1c0f032a56fe13aa2abce1ee97fd2cb0aafec32efecdcb27aec" gracePeriod=25 Oct 31 21:27:07.676078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.676040 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb"] Oct 31 21:27:07.676836 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.676811 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" Oct 31 21:27:07.676836 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.676835 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" Oct 31 21:27:07.677037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.676855 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" Oct 31 21:27:07.677037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.676863 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" Oct 31 21:27:07.677037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.676990 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" Oct 31 21:27:07.677037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.677003 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerName="istio-proxy" Oct 31 21:27:07.681151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.681124 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.695954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.695925 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-994336-openshift-default-dockercfg-qgxk4\"" Oct 31 21:27:07.708067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.708038 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb"] Oct 31 21:27:07.759691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.759657 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.759866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.759700 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca228a8-ae18-4f21-8238-a23e53143a5e-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.759866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.759727 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.759866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.759779 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.759866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.759820 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.759866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.759859 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.760031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.759877 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.760031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.759902 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hthc4\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-kube-api-access-hthc4\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.760031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.759930 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.789456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.785209 5188 generic.go:356] "Generic (PLEG): container finished" podID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" containerID="762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb" exitCode=0 Oct 31 21:27:07.789456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.785342 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" event={"ID":"7bc98e0d-0445-4a04-b924-8da163c7f7e8","Type":"ContainerDied","Data":"762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb"} Oct 31 21:27:07.789456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.785413 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" event={"ID":"7bc98e0d-0445-4a04-b924-8da163c7f7e8","Type":"ContainerDied","Data":"8495a84cbbb8330e497adf8ac8fba61234716af27c2ae238112f539cd4bb9bfb"} Oct 31 21:27:07.789456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.785447 5188 scope.go:122] "RemoveContainer" containerID="762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb" Oct 31 21:27:07.789456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.785632 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr" Oct 31 21:27:07.796978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.790679 5188 generic.go:356] "Generic (PLEG): container finished" podID="9de40230-111e-4929-b464-7a6a11a62310" containerID="a5c952172b60b1c0f032a56fe13aa2abce1ee97fd2cb0aafec32efecdcb27aec" exitCode=0 Oct 31 21:27:07.796978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.790787 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" event={"ID":"9de40230-111e-4929-b464-7a6a11a62310","Type":"ContainerDied","Data":"a5c952172b60b1c0f032a56fe13aa2abce1ee97fd2cb0aafec32efecdcb27aec"} Oct 31 21:27:07.796978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.793044 5188 generic.go:356] "Generic (PLEG): container finished" podID="6132851e-aa58-4729-899c-9421e921879a" containerID="6ff6188b70c0a00b2bae8f12a20da62557e61dff6fc57e38819a756195c07356" exitCode=0 Oct 31 21:27:07.796978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.793950 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" event={"ID":"6132851e-aa58-4729-899c-9421e921879a","Type":"ContainerDied","Data":"6ff6188b70c0a00b2bae8f12a20da62557e61dff6fc57e38819a756195c07356"} Oct 31 21:27:07.853929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.853902 5188 scope.go:122] "RemoveContainer" containerID="c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076" Oct 31 21:27:07.861221 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.861189 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.861720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.861616 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.862277 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.862256 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.862612 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.862588 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.862858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.862837 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.863226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.863207 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.863386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.863242 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.863530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.863511 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.863636 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.863523 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hthc4\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-kube-api-access-hthc4\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.863770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.863752 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.863912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.863896 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.864031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.864014 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca228a8-ae18-4f21-8238-a23e53143a5e-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.864172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.863789 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.865167 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.865102 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca228a8-ae18-4f21-8238-a23e53143a5e-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.866836 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.866802 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.868461 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.868425 5188 scope.go:122] "RemoveContainer" containerID="762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb" Oct 31 21:27:07.868885 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:27:07.868764 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb\": container with ID starting with 762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb not found: ID does not exist" containerID="762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb" Oct 31 21:27:07.868885 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.868801 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb"} err="failed to get container status \"762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb\": rpc error: code = NotFound desc = could not find container \"762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb\": container with ID starting with 762ed7d2876ac7e2c2579a8b46a3464fb3f4e30650e93d0a41688c5a71f953cb not found: ID does not exist" Oct 31 21:27:07.868885 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.868828 5188 scope.go:122] "RemoveContainer" containerID="c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076" Oct 31 21:27:07.869182 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:27:07.869146 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076\": container with ID starting with c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076 not found: ID does not exist" containerID="c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076" Oct 31 21:27:07.869289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.869174 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076"} err="failed to get container status \"c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076\": rpc error: code = NotFound desc = could not find container \"c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076\": container with ID starting with c90749569e330f153ff932e97ebe9ebf9ccdc1b47d2e9637aba4ffdf54844076 not found: ID does not exist" Oct 31 21:27:07.877844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.877013 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.881532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.881510 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr"] Oct 31 21:27:07.884911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.884865 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.886907 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.886882 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-25447e22-openshift-default-56df68cbf9-5tstr"] Oct 31 21:27:07.888629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.888597 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hthc4\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-kube-api-access-hthc4\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:07.926281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.926240 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:27:07.965379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.965347 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6132851e-aa58-4729-899c-9421e921879a-kube-api-access-n2484\" (UniqueName: \"kubernetes.io/projected/6132851e-aa58-4729-899c-9421e921879a-kube-api-access-n2484\") pod \"6132851e-aa58-4729-899c-9421e921879a\" (UID: \"6132851e-aa58-4729-899c-9421e921879a\") " Oct 31 21:27:07.965564 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.965500 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6132851e-aa58-4729-899c-9421e921879a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6132851e-aa58-4729-899c-9421e921879a-plugins-config-volume\") pod \"6132851e-aa58-4729-899c-9421e921879a\" (UID: \"6132851e-aa58-4729-899c-9421e921879a\") " Oct 31 21:27:07.965915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.965889 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6132851e-aa58-4729-899c-9421e921879a-plugins-config-volume" pod "6132851e-aa58-4729-899c-9421e921879a" (UID: "6132851e-aa58-4729-899c-9421e921879a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:27:07.967744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.967720 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6132851e-aa58-4729-899c-9421e921879a-kube-api-access-n2484" pod "6132851e-aa58-4729-899c-9421e921879a" (UID: "6132851e-aa58-4729-899c-9421e921879a"). InnerVolumeSpecName "kube-api-access-n2484". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:07.969652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:07.969631 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:27:08.000938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.000899 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:08.066801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.066765 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de40230-111e-4929-b464-7a6a11a62310-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de40230-111e-4929-b464-7a6a11a62310-plugins-config-volume\") pod \"9de40230-111e-4929-b464-7a6a11a62310\" (UID: \"9de40230-111e-4929-b464-7a6a11a62310\") " Oct 31 21:27:08.066997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.066901 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de40230-111e-4929-b464-7a6a11a62310-kube-api-access-hnpc6\" (UniqueName: \"kubernetes.io/projected/9de40230-111e-4929-b464-7a6a11a62310-kube-api-access-hnpc6\") pod \"9de40230-111e-4929-b464-7a6a11a62310\" (UID: \"9de40230-111e-4929-b464-7a6a11a62310\") " Oct 31 21:27:08.067120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.067106 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2484\" (UniqueName: \"kubernetes.io/projected/6132851e-aa58-4729-899c-9421e921879a-kube-api-access-n2484\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:08.067184 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.067120 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6132851e-aa58-4729-899c-9421e921879a-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:08.067442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.067416 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de40230-111e-4929-b464-7a6a11a62310-plugins-config-volume" pod "9de40230-111e-4929-b464-7a6a11a62310" (UID: "9de40230-111e-4929-b464-7a6a11a62310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:27:08.068964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.068939 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de40230-111e-4929-b464-7a6a11a62310-kube-api-access-hnpc6" pod "9de40230-111e-4929-b464-7a6a11a62310" (UID: "9de40230-111e-4929-b464-7a6a11a62310"). InnerVolumeSpecName "kube-api-access-hnpc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:08.167961 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.167869 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpc6\" (UniqueName: \"kubernetes.io/projected/9de40230-111e-4929-b464-7a6a11a62310-kube-api-access-hnpc6\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:08.167961 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.167905 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9de40230-111e-4929-b464-7a6a11a62310-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:08.374073 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.374000 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb"] Oct 31 21:27:08.377502 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:27:08.377469 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca228a8_ae18_4f21_8238_a23e53143a5e.slice/crio-d5775470930f17aecf876df003938f4a95f3421e3f1a5fa60840dbfaa70f5305 WatchSource:0}: Error finding container d5775470930f17aecf876df003938f4a95f3421e3f1a5fa60840dbfaa70f5305: Status 404 returned error can't find the container with id d5775470930f17aecf876df003938f4a95f3421e3f1a5fa60840dbfaa70f5305 Oct 31 21:27:08.379966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.379938 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:08.380046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.380001 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:08.380046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.380034 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:08.803407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.803302 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" Oct 31 21:27:08.803407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.803334 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr" event={"ID":"9de40230-111e-4929-b464-7a6a11a62310","Type":"ContainerDied","Data":"c39f4bd903c44ce73704036cb9cdc10487e67290d488af020067a97f746e0da1"} Oct 31 21:27:08.803929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.803410 5188 scope.go:122] "RemoveContainer" containerID="a5c952172b60b1c0f032a56fe13aa2abce1ee97fd2cb0aafec32efecdcb27aec" Oct 31 21:27:08.804747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.804717 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" event={"ID":"6132851e-aa58-4729-899c-9421e921879a","Type":"ContainerDied","Data":"04079e0191fa357e763bfafb02912d18e94e939ee8bf4f53566f8d64779bdb2e"} Oct 31 21:27:08.804823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.804740 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc" Oct 31 21:27:08.807346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.807221 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" event={"ID":"5ca228a8-ae18-4f21-8238-a23e53143a5e","Type":"ContainerStarted","Data":"0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826"} Oct 31 21:27:08.807346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.807272 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" event={"ID":"5ca228a8-ae18-4f21-8238-a23e53143a5e","Type":"ContainerStarted","Data":"d5775470930f17aecf876df003938f4a95f3421e3f1a5fa60840dbfaa70f5305"} Oct 31 21:27:08.819023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.818999 5188 scope.go:122] "RemoveContainer" containerID="6ff6188b70c0a00b2bae8f12a20da62557e61dff6fc57e38819a756195c07356" Oct 31 21:27:08.871498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.871459 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr"] Oct 31 21:27:08.871693 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.871524 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7ef0e96a-7c8b45947-vkvqr"] Oct 31 21:27:08.908276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.908238 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc"] Oct 31 21:27:08.915115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.915081 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7ef0e96a-7b4d7b87c7-gv4rc"] Oct 31 21:27:08.961373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:08.961276 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" podStartSLOduration=1.961257657 podStartE2EDuration="1.961257657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:08.959377394 +0000 UTC m=+675.652334784" watchObservedRunningTime="2026-10-31 21:27:08.961257657 +0000 UTC m=+675.654215115" Oct 31 21:27:09.001189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:09.001153 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:09.008523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:09.008306 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:09.472661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:09.472624 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6132851e-aa58-4729-899c-9421e921879a" path="/var/lib/kubelet/pods/6132851e-aa58-4729-899c-9421e921879a/volumes" Oct 31 21:27:09.473076 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:09.473062 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc98e0d-0445-4a04-b924-8da163c7f7e8" path="/var/lib/kubelet/pods/7bc98e0d-0445-4a04-b924-8da163c7f7e8/volumes" Oct 31 21:27:09.473417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:09.473405 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de40230-111e-4929-b464-7a6a11a62310" path="/var/lib/kubelet/pods/9de40230-111e-4929-b464-7a6a11a62310/volumes" Oct 31 21:27:09.816149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:09.816072 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:09.817123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:09.817103 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:11.108385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.108348 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9"] Oct 31 21:27:11.108858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.108821 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9de40230-111e-4929-b464-7a6a11a62310" containerName="payload-pre-processing" Oct 31 21:27:11.108858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.108832 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de40230-111e-4929-b464-7a6a11a62310" containerName="payload-pre-processing" Oct 31 21:27:11.108858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.108850 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6132851e-aa58-4729-899c-9421e921879a" containerName="payload-processing" Oct 31 21:27:11.108858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.108855 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6132851e-aa58-4729-899c-9421e921879a" containerName="payload-processing" Oct 31 21:27:11.108989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.108923 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6132851e-aa58-4729-899c-9421e921879a" containerName="payload-processing" Oct 31 21:27:11.108989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.108935 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de40230-111e-4929-b464-7a6a11a62310" containerName="payload-pre-processing" Oct 31 21:27:11.113916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.113891 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:11.137248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.137200 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9"] Oct 31 21:27:11.145922 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.145874 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-cc45a7\"" Oct 31 21:27:11.146133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.146100 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-cc45a7-dockercfg-6qh7z\"" Oct 31 21:27:11.206624 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.206585 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25trr\" (UniqueName: \"kubernetes.io/projected/145a082c-6551-481b-864f-703dacdcd332-kube-api-access-25trr\") pod \"payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9\" (UID: \"145a082c-6551-481b-864f-703dacdcd332\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:11.206829 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.206716 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145a082c-6551-481b-864f-703dacdcd332-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9\" (UID: \"145a082c-6551-481b-864f-703dacdcd332\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:11.307480 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.307435 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145a082c-6551-481b-864f-703dacdcd332-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9\" (UID: \"145a082c-6551-481b-864f-703dacdcd332\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:11.307708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.307643 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25trr\" (UniqueName: \"kubernetes.io/projected/145a082c-6551-481b-864f-703dacdcd332-kube-api-access-25trr\") pod \"payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9\" (UID: \"145a082c-6551-481b-864f-703dacdcd332\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:11.308249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.308227 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145a082c-6551-481b-864f-703dacdcd332-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9\" (UID: \"145a082c-6551-481b-864f-703dacdcd332\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:11.348757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.348724 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25trr\" (UniqueName: \"kubernetes.io/projected/145a082c-6551-481b-864f-703dacdcd332-kube-api-access-25trr\") pod \"payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9\" (UID: \"145a082c-6551-481b-864f-703dacdcd332\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:11.428995 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.428967 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:11.711331 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.711288 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:27:11.758308 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.758276 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9"] Oct 31 21:27:11.814857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.814758 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-envoy\") pod \"063ca5f5-b558-47b2-9960-bdebb49d83a3\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " Oct 31 21:27:11.814857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.814808 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063ca5f5-b558-47b2-9960-bdebb49d83a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063ca5f5-b558-47b2-9960-bdebb49d83a3-istiod-ca-cert\") pod \"063ca5f5-b558-47b2-9960-bdebb49d83a3\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " Oct 31 21:27:11.815039 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.814856 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-token\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-token\") pod \"063ca5f5-b558-47b2-9960-bdebb49d83a3\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " Oct 31 21:27:11.815039 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.814911 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-data\") pod \"063ca5f5-b558-47b2-9960-bdebb49d83a3\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " Oct 31 21:27:11.815039 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.814948 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-socket\") pod \"063ca5f5-b558-47b2-9960-bdebb49d83a3\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " Oct 31 21:27:11.815039 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.814997 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-podinfo\") pod \"063ca5f5-b558-47b2-9960-bdebb49d83a3\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " Oct 31 21:27:11.815254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.815128 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-kube-api-access-knpnm\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-kube-api-access-knpnm\") pod \"063ca5f5-b558-47b2-9960-bdebb49d83a3\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " Oct 31 21:27:11.815254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.815162 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-certs\") pod \"063ca5f5-b558-47b2-9960-bdebb49d83a3\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " Oct 31 21:27:11.815254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.815204 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-credential-socket\") pod \"063ca5f5-b558-47b2-9960-bdebb49d83a3\" (UID: \"063ca5f5-b558-47b2-9960-bdebb49d83a3\") " Oct 31 21:27:11.815920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.815707 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-data" pod "063ca5f5-b558-47b2-9960-bdebb49d83a3" (UID: "063ca5f5-b558-47b2-9960-bdebb49d83a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:11.815920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.815798 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-credential-socket" pod "063ca5f5-b558-47b2-9960-bdebb49d83a3" (UID: "063ca5f5-b558-47b2-9960-bdebb49d83a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:11.816126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.815986 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-certs" pod "063ca5f5-b558-47b2-9960-bdebb49d83a3" (UID: "063ca5f5-b558-47b2-9960-bdebb49d83a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:11.816360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.816340 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-socket" pod "063ca5f5-b558-47b2-9960-bdebb49d83a3" (UID: "063ca5f5-b558-47b2-9960-bdebb49d83a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:11.816490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.816391 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063ca5f5-b558-47b2-9960-bdebb49d83a3-istiod-ca-cert" pod "063ca5f5-b558-47b2-9960-bdebb49d83a3" (UID: "063ca5f5-b558-47b2-9960-bdebb49d83a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:27:11.817700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.817675 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-envoy" pod "063ca5f5-b558-47b2-9960-bdebb49d83a3" (UID: "063ca5f5-b558-47b2-9960-bdebb49d83a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:27:11.817962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.817928 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-kube-api-access-knpnm" pod "063ca5f5-b558-47b2-9960-bdebb49d83a3" (UID: "063ca5f5-b558-47b2-9960-bdebb49d83a3"). InnerVolumeSpecName "kube-api-access-knpnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:11.817962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.817953 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-podinfo" pod "063ca5f5-b558-47b2-9960-bdebb49d83a3" (UID: "063ca5f5-b558-47b2-9960-bdebb49d83a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:27:11.818194 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.818102 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-token" pod "063ca5f5-b558-47b2-9960-bdebb49d83a3" (UID: "063ca5f5-b558-47b2-9960-bdebb49d83a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:11.831747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.831712 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" event={"ID":"145a082c-6551-481b-864f-703dacdcd332","Type":"ContainerStarted","Data":"6fbd22ee9b4a70db7425edbcaab7cd6099240cf4690ec4d320f4b6ee9775faa9"} Oct 31 21:27:11.834041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.834010 5188 generic.go:356] "Generic (PLEG): container finished" podID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerID="c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d" exitCode=0 Oct 31 21:27:11.834169 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.834132 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" Oct 31 21:27:11.834267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.834249 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" event={"ID":"063ca5f5-b558-47b2-9960-bdebb49d83a3","Type":"ContainerDied","Data":"c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d"} Oct 31 21:27:11.834374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.834301 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q" event={"ID":"063ca5f5-b558-47b2-9960-bdebb49d83a3","Type":"ContainerDied","Data":"60dfefcad81d00200546d642b68d84a02fa1124dac8cc09e76fcf0b7b2962472"} Oct 31 21:27:11.835348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.835306 5188 scope.go:122] "RemoveContainer" containerID="c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d" Oct 31 21:27:11.874904 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.874873 5188 scope.go:122] "RemoveContainer" containerID="729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2" Oct 31 21:27:11.888234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.888203 5188 scope.go:122] "RemoveContainer" containerID="c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d" Oct 31 21:27:11.888615 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:27:11.888591 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d\": container with ID starting with c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d not found: ID does not exist" containerID="c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d" Oct 31 21:27:11.888723 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.888626 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d"} err="failed to get container status \"c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d\": rpc error: code = NotFound desc = could not find container \"c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d\": container with ID starting with c77352933b7b815560ebf65ead6203765dffd5d3dde5a71480381e78a37f312d not found: ID does not exist" Oct 31 21:27:11.888723 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.888651 5188 scope.go:122] "RemoveContainer" containerID="729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2" Oct 31 21:27:11.888921 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:27:11.888899 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2\": container with ID starting with 729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2 not found: ID does not exist" containerID="729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2" Oct 31 21:27:11.889034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.888931 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2"} err="failed to get container status \"729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2\": rpc error: code = NotFound desc = could not find container \"729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2\": container with ID starting with 729ba932728b2d813137077cd6e011f858c1fcaee360851508eaf77b0e7e97f2 not found: ID does not exist" Oct 31 21:27:11.914080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.914033 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q"] Oct 31 21:27:11.916847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.916804 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:11.916847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.916839 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:11.916847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.916853 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:11.917154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.916866 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:11.917154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.916878 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knpnm\" (UniqueName: \"kubernetes.io/projected/063ca5f5-b558-47b2-9960-bdebb49d83a3-kube-api-access-knpnm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:11.917154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.916892 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:11.917154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.916905 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:11.917154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.916916 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063ca5f5-b558-47b2-9960-bdebb49d83a3-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:11.917154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.916928 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063ca5f5-b558-47b2-9960-bdebb49d83a3-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:27:11.935248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:11.935207 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-7ef0e96a-openshift-default-54cf9cd8bf-rng5q"] Oct 31 21:27:12.842700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:12.842658 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" event={"ID":"145a082c-6551-481b-864f-703dacdcd332","Type":"ContainerStarted","Data":"ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18"} Oct 31 21:27:12.843167 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:12.842762 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:12.885297 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:12.885230 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" podStartSLOduration=1.885210662 podStartE2EDuration="1.885210662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:12.884299755 +0000 UTC m=+679.577257147" watchObservedRunningTime="2026-10-31 21:27:12.885210662 +0000 UTC m=+679.578168055" Oct 31 21:27:13.472380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:13.472345 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" path="/var/lib/kubelet/pods/063ca5f5-b558-47b2-9960-bdebb49d83a3/volumes" Oct 31 21:27:14.984309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:14.984267 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz"] Oct 31 21:27:14.984773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:14.984759 5188 cpu_manager.go:414] "containerMap: removing container" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" Oct 31 21:27:14.984817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:14.984774 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" Oct 31 21:27:14.984817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:14.984804 5188 cpu_manager.go:414] "containerMap: removing container" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" Oct 31 21:27:14.984817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:14.984810 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" Oct 31 21:27:14.984921 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:14.984880 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" Oct 31 21:27:14.984921 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:14.984891 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="063ca5f5-b558-47b2-9960-bdebb49d83a3" containerName="istio-proxy" Oct 31 21:27:14.989703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:14.989678 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:15.007266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.007239 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-994336-dockercfg-x6c29\"" Oct 31 21:27:15.007490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.007477 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-994336\"" Oct 31 21:27:15.020217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.020183 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz"] Oct 31 21:27:15.065691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.065656 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt"] Oct 31 21:27:15.069893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.069847 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:15.091368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.090755 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt"] Oct 31 21:27:15.153029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.152994 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f266a-8ca0-4e78-9e23-064ccc6e761c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz\" (UID: \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:15.153029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.153033 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblsx\" (UniqueName: \"kubernetes.io/projected/b68f266a-8ca0-4e78-9e23-064ccc6e761c-kube-api-access-rblsx\") pod \"payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz\" (UID: \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:15.254624 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.254515 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt\" (UID: \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:15.254780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.254637 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6l9\" (UniqueName: \"kubernetes.io/projected/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-kube-api-access-xf6l9\") pod \"payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt\" (UID: \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:15.254780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.254711 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f266a-8ca0-4e78-9e23-064ccc6e761c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz\" (UID: \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:15.254780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.254737 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblsx\" (UniqueName: \"kubernetes.io/projected/b68f266a-8ca0-4e78-9e23-064ccc6e761c-kube-api-access-rblsx\") pod \"payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz\" (UID: \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:15.255517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.255498 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f266a-8ca0-4e78-9e23-064ccc6e761c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz\" (UID: \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:15.279204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.279164 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblsx\" (UniqueName: \"kubernetes.io/projected/b68f266a-8ca0-4e78-9e23-064ccc6e761c-kube-api-access-rblsx\") pod \"payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz\" (UID: \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:15.307179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.307144 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:15.355796 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.355759 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt\" (UID: \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:15.355990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.355823 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6l9\" (UniqueName: \"kubernetes.io/projected/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-kube-api-access-xf6l9\") pod \"payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt\" (UID: \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:15.356603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.356580 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt\" (UID: \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:15.385351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.384471 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6l9\" (UniqueName: \"kubernetes.io/projected/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-kube-api-access-xf6l9\") pod \"payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt\" (UID: \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:15.432525 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.432485 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:15.656310 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.656276 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz"] Oct 31 21:27:15.658466 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:27:15.658436 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68f266a_8ca0_4e78_9e23_064ccc6e761c.slice/crio-9dc9738a4c9e075fbb920309c411391f33dcc95a9d844462bc7a1432d74cc328 WatchSource:0}: Error finding container 9dc9738a4c9e075fbb920309c411391f33dcc95a9d844462bc7a1432d74cc328: Status 404 returned error can't find the container with id 9dc9738a4c9e075fbb920309c411391f33dcc95a9d844462bc7a1432d74cc328 Oct 31 21:27:15.756598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.756401 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt"] Oct 31 21:27:15.756710 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:27:15.756611 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc6ae49_47d0_4557_8f0e_7a8b32f3b3c9.slice/crio-dd0638baeeee0218bec88365670fe0de436af09b7545bdec98d80fce8f6bb56a WatchSource:0}: Error finding container dd0638baeeee0218bec88365670fe0de436af09b7545bdec98d80fce8f6bb56a: Status 404 returned error can't find the container with id dd0638baeeee0218bec88365670fe0de436af09b7545bdec98d80fce8f6bb56a Oct 31 21:27:15.875485 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.875437 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" event={"ID":"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9","Type":"ContainerStarted","Data":"c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7"} Oct 31 21:27:15.875690 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.875545 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" event={"ID":"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9","Type":"ContainerStarted","Data":"dd0638baeeee0218bec88365670fe0de436af09b7545bdec98d80fce8f6bb56a"} Oct 31 21:27:15.875690 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.875587 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:15.877490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.877457 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" event={"ID":"b68f266a-8ca0-4e78-9e23-064ccc6e761c","Type":"ContainerStarted","Data":"13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5"} Oct 31 21:27:15.877627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.877504 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" event={"ID":"b68f266a-8ca0-4e78-9e23-064ccc6e761c","Type":"ContainerStarted","Data":"9dc9738a4c9e075fbb920309c411391f33dcc95a9d844462bc7a1432d74cc328"} Oct 31 21:27:15.877627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.877592 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:15.931484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:15.931411 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" podStartSLOduration=0.931389847 podStartE2EDuration="931.389847ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:15.928815606 +0000 UTC m=+682.621772992" watchObservedRunningTime="2026-10-31 21:27:15.931389847 +0000 UTC m=+682.624347239" Oct 31 21:27:17.799628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:17.799598 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 21:27:17.838229 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:17.838176 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" podStartSLOduration=3.8381632039999998 podStartE2EDuration="3.838163204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:15.992784034 +0000 UTC m=+682.685741427" watchObservedRunningTime="2026-10-31 21:27:17.838163204 +0000 UTC m=+684.531120588" Oct 31 21:27:19.456224 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.456192 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m"] Oct 31 21:27:19.461992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.461959 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.487757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.486608 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b5989\"/\"openshift-service-ca.crt\"" Oct 31 21:27:19.508117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.504375 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b5989\"/\"default-dockercfg-tdnws\"" Oct 31 21:27:19.508117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.504456 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b5989\"/\"kube-root-ca.crt\"" Oct 31 21:27:19.508117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.506598 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b5989\"/\"simulator-w0-4b5989-kserve-self-signed-certs\"" Oct 31 21:27:19.508366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.508195 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m"] Oct 31 21:27:19.519103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.519064 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tmp-dir\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.519284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.519175 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-model-cache\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.519284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.519216 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-home\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.519284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.519243 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-dshm\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.519284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.519271 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrgk\" (UniqueName: \"kubernetes.io/projected/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-kube-api-access-xwrgk\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.519533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.519336 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tls-certs\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.620217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.620176 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tmp-dir\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.620402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.620317 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-model-cache\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.620402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.620387 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-home\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.620535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.620404 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-dshm\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.620535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.620422 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrgk\" (UniqueName: \"kubernetes.io/projected/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-kube-api-access-xwrgk\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.620535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.620457 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tls-certs\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.620719 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.620691 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tmp-dir\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.620784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.620707 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-model-cache\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.621094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.621070 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-home\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.622730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.622708 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-dshm\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.622824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.622807 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tls-certs\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.641181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.641148 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrgk\" (UniqueName: \"kubernetes.io/projected/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-kube-api-access-xwrgk\") pod \"simulator-w0-4b5989-kserve-ddf7b6758-7ld8m\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:19.810014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:19.809868 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:20.112886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:20.112864 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m"] Oct 31 21:27:20.114993 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:27:20.114958 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b18e0f7_04d4_4854_b47b_27b07dde1b1b.slice/crio-d87f2527eb1dc15fb28bc894f6dd4002366ecd68f76a17bd0d0051bf1fe417db WatchSource:0}: Error finding container d87f2527eb1dc15fb28bc894f6dd4002366ecd68f76a17bd0d0051bf1fe417db: Status 404 returned error can't find the container with id d87f2527eb1dc15fb28bc894f6dd4002366ecd68f76a17bd0d0051bf1fe417db Oct 31 21:27:20.933405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:20.933355 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" event={"ID":"6b18e0f7-04d4-4854-b47b-27b07dde1b1b","Type":"ContainerStarted","Data":"cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38"} Oct 31 21:27:20.933835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:20.933415 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" event={"ID":"6b18e0f7-04d4-4854-b47b-27b07dde1b1b","Type":"ContainerStarted","Data":"d87f2527eb1dc15fb28bc894f6dd4002366ecd68f76a17bd0d0051bf1fe417db"} Oct 31 21:27:20.996387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:20.973091 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podStartSLOduration=1.9730694519999998 podStartE2EDuration="1.973069452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:20.971679285 +0000 UTC m=+687.664636678" watchObservedRunningTime="2026-10-31 21:27:20.973069452 +0000 UTC m=+687.666026845" Oct 31 21:27:21.818994 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:21.818958 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:23.855571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:23.855529 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 21:27:26.888051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:26.888020 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 21:27:26.888628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:26.888089 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 21:27:26.997410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:26.997371 5188 generic.go:356] "Generic (PLEG): container finished" podID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerID="2205f7189c45a4ae05943c1a68ec455137ef1cbfcc0f9eaa38e43cbb9e58760c" exitCode=0 Oct 31 21:27:26.997800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:26.997776 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" event={"ID":"2829db12-be6a-46ba-a4d9-58158af0f3b4","Type":"ContainerDied","Data":"2205f7189c45a4ae05943c1a68ec455137ef1cbfcc0f9eaa38e43cbb9e58760c"} Oct 31 21:27:26.998403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:26.998383 5188 scope.go:122] "RemoveContainer" containerID="2205f7189c45a4ae05943c1a68ec455137ef1cbfcc0f9eaa38e43cbb9e58760c" Oct 31 21:27:27.001064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:27.001029 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:27.001279 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:27.001261 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:27.001441 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:27.001422 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:28.008186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:28.008136 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" event={"ID":"2829db12-be6a-46ba-a4d9-58158af0f3b4","Type":"ContainerStarted","Data":"0e36e3cc2c63c78607c6637f75b461a368c97cfc1d52e518b199d19647a6cfc6"} Oct 31 21:27:28.601349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:28.600760 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:28.616348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:28.609480 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:27:28.616348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:28.609546 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 21:27:29.595348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:29.593483 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:27:29.595348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:29.593567 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 21:27:29.814348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:29.814061 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:29.833552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:29.833523 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:30.046277 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:30.046239 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:27:30.627349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:30.624504 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:27:30.627349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:30.624579 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 21:27:31.593350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:31.593023 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:27:31.593350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:31.593095 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 21:27:32.561133 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:27:32.561082 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:27:32.598350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:32.594738 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:27:32.598350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:32.594800 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 21:27:33.603402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:33.603365 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:33.603853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:33.603837 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:33.604856 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:33.604834 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:27:37.085870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:37.085830 5188 generic.go:356] "Generic (PLEG): container finished" podID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerID="0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826" exitCode=0 Oct 31 21:27:37.086318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:37.085906 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" event={"ID":"5ca228a8-ae18-4f21-8238-a23e53143a5e","Type":"ContainerDied","Data":"0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826"} Oct 31 21:27:37.086318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:37.086347 5188 scope.go:122] "RemoveContainer" containerID="0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826" Oct 31 21:27:37.087645 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:37.087619 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:37.087719 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:37.087690 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:37.087755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:37.087730 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:27:38.108212 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:38.108173 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" event={"ID":"5ca228a8-ae18-4f21-8238-a23e53143a5e","Type":"ContainerStarted","Data":"a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34"} Oct 31 21:27:39.002533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:39.002359 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:39.003745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:39.003700 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 21:27:39.003919 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:39.003765 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 21:27:39.816041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:39.816001 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:40.002798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:40.002754 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 21:27:40.003019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:40.002811 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 21:27:41.007348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:41.005787 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 21:27:41.007348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:41.005868 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 21:27:41.936117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:41.936078 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v"] Oct 31 21:27:41.940492 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:41.940465 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:41.968215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:41.968186 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc45a7\"/\"openshift-service-ca.crt\"" Oct 31 21:27:41.987349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:41.980659 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc45a7\"/\"kube-root-ca.crt\"" Oct 31 21:27:41.987349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:41.980835 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc45a7\"/\"simulator-w4-cc45a7-kserve-self-signed-certs\"" Oct 31 21:27:41.987349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:41.980946 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc45a7\"/\"default-dockercfg-v4bmc\"" Oct 31 21:27:42.002853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.002804 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 21:27:42.003160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.003127 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 21:27:42.008694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.008591 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v"] Oct 31 21:27:42.040077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.040043 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-dshm\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.040273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.040119 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djl94\" (UniqueName: \"kubernetes.io/projected/3f4b299a-da61-44bd-ae6f-5da0499400c8-kube-api-access-djl94\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.040273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.040150 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-home\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.040273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.040249 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4b299a-da61-44bd-ae6f-5da0499400c8-tls-certs\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.040412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.040279 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-tmp-dir\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.040412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.040295 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-model-cache\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.140934 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.140885 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4b299a-da61-44bd-ae6f-5da0499400c8-tls-certs\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.141219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.140949 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-tmp-dir\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.141219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.140989 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-model-cache\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.141219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.141039 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-dshm\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.141219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.141111 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djl94\" (UniqueName: \"kubernetes.io/projected/3f4b299a-da61-44bd-ae6f-5da0499400c8-kube-api-access-djl94\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.141219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.141155 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-home\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.141527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.141482 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-tmp-dir\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.141583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.141548 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-model-cache\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.141691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.141657 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-home\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.143566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.143542 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-dshm\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.143846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.143829 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4b299a-da61-44bd-ae6f-5da0499400c8-tls-certs\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.163225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.163189 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djl94\" (UniqueName: \"kubernetes.io/projected/3f4b299a-da61-44bd-ae6f-5da0499400c8-kube-api-access-djl94\") pod \"simulator-w4-cc45a7-kserve-68b458c76d-2nr8v\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.253717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.253622 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:42.599821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:42.599720 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v"] Oct 31 21:27:43.002415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:43.002374 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 21:27:43.002591 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:43.002448 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 21:27:43.151810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:43.151772 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" event={"ID":"3f4b299a-da61-44bd-ae6f-5da0499400c8","Type":"ContainerStarted","Data":"7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38"} Oct 31 21:27:43.152221 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:43.151819 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" event={"ID":"3f4b299a-da61-44bd-ae6f-5da0499400c8","Type":"ContainerStarted","Data":"cd90046403a856420d4de7a1b7a1dda5746bf794b2144ffc4bd5a3b1d76b9755"} Oct 31 21:27:43.190043 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:43.189975 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podStartSLOduration=2.189961302 podStartE2EDuration="2.189961302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:43.18884189 +0000 UTC m=+709.881799282" watchObservedRunningTime="2026-10-31 21:27:43.189961302 +0000 UTC m=+709.882918693" Oct 31 21:27:43.253857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:43.253771 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:44.006102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:44.006069 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:44.007197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:44.007174 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:27:52.254174 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.254137 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:52.266803 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.266776 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:52.887762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.887724 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4"] Oct 31 21:27:52.900774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.900740 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:52.910355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.910296 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4"] Oct 31 21:27:52.916977 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.916945 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b5989\"/\"premium-w0-4b5989-kserve-self-signed-certs\"" Oct 31 21:27:52.948092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.948052 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-dshm\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:52.948092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.948090 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74pc\" (UniqueName: \"kubernetes.io/projected/c1176d9d-4701-4c7d-84ec-60f664b45470-kube-api-access-l74pc\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:52.948312 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.948181 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-home\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:52.948312 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.948236 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-model-cache\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:52.948312 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.948281 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-tmp-dir\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:52.948450 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:52.948421 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1176d9d-4701-4c7d-84ec-60f664b45470-tls-certs\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.049813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.049777 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1176d9d-4701-4c7d-84ec-60f664b45470-tls-certs\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.049996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.049834 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-dshm\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.049996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.049850 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74pc\" (UniqueName: \"kubernetes.io/projected/c1176d9d-4701-4c7d-84ec-60f664b45470-kube-api-access-l74pc\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.049996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.049884 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-home\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.049996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.049906 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-model-cache\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.050195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.050120 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-tmp-dir\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.050376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.050355 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-model-cache\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.050475 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.050431 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-home\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.050534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.050511 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-tmp-dir\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.052075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.052056 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-dshm\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.052369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.052349 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1176d9d-4701-4c7d-84ec-60f664b45470-tls-certs\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.075720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.073099 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74pc\" (UniqueName: \"kubernetes.io/projected/c1176d9d-4701-4c7d-84ec-60f664b45470-kube-api-access-l74pc\") pod \"premium-w0-4b5989-kserve-dd687bf47-qtwv4\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.220486 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.220376 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:27:53.239031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.239002 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:27:53.529374 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:27:53.529340 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1176d9d_4701_4c7d_84ec_60f664b45470.slice/crio-6c0c009e7c329ccf444121d651ec71205d1d4396c681e9c9466edbf2874accdd WatchSource:0}: Error finding container 6c0c009e7c329ccf444121d651ec71205d1d4396c681e9c9466edbf2874accdd: Status 404 returned error can't find the container with id 6c0c009e7c329ccf444121d651ec71205d1d4396c681e9c9466edbf2874accdd Oct 31 21:27:53.529374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:53.529360 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4"] Oct 31 21:27:54.231758 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:54.231718 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" event={"ID":"c1176d9d-4701-4c7d-84ec-60f664b45470","Type":"ContainerStarted","Data":"c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771"} Oct 31 21:27:54.231758 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:54.231764 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" event={"ID":"c1176d9d-4701-4c7d-84ec-60f664b45470","Type":"ContainerStarted","Data":"6c0c009e7c329ccf444121d651ec71205d1d4396c681e9c9466edbf2874accdd"} Oct 31 21:27:54.269966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:54.269911 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podStartSLOduration=2.269897204 podStartE2EDuration="2.269897204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:54.267729213 +0000 UTC m=+720.960686604" watchObservedRunningTime="2026-10-31 21:27:54.269897204 +0000 UTC m=+720.962854614" Oct 31 21:27:55.220756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:27:55.220714 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:28:02.537022 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:28:02.536983 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:28:03.221262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:03.221225 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:28:03.234611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:03.234573 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:28:03.303072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:03.303043 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:28:04.333482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.333450 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd"] Oct 31 21:28:04.339851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.339821 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.366259 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.366225 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"default-dockercfg-g8p4t\"" Oct 31 21:28:04.366259 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.366249 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"openshift-service-ca.crt\"" Oct 31 21:28:04.366515 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.366376 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"kube-root-ca.crt\"" Oct 31 21:28:04.366515 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.366379 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"premium-w3-106555-kserve-self-signed-certs\"" Oct 31 21:28:04.369214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.369188 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd"] Oct 31 21:28:04.474077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.474043 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-tmp-dir\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.474077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.474090 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-model-cache\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.474317 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.474110 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w9q9\" (UniqueName: \"kubernetes.io/projected/9e0bae2f-c86a-487e-ac13-6b7890092979-kube-api-access-4w9q9\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.474317 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.474234 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0bae2f-c86a-487e-ac13-6b7890092979-tls-certs\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.474432 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.474379 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-home\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.474503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.474485 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-dshm\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.575915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.575876 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-home\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.576120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.575956 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-dshm\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.576120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.575981 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-tmp-dir\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.576120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.576008 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-model-cache\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.576120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.576031 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w9q9\" (UniqueName: \"kubernetes.io/projected/9e0bae2f-c86a-487e-ac13-6b7890092979-kube-api-access-4w9q9\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.576120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.576083 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0bae2f-c86a-487e-ac13-6b7890092979-tls-certs\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.576512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.576483 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-tmp-dir\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.576616 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.576592 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-model-cache\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.576687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.576670 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-home\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.578469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.578447 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-dshm\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.578841 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.578819 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0bae2f-c86a-487e-ac13-6b7890092979-tls-certs\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.597898 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.597830 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w9q9\" (UniqueName: \"kubernetes.io/projected/9e0bae2f-c86a-487e-ac13-6b7890092979-kube-api-access-4w9q9\") pod \"premium-w3-106555-kserve-7ff9cb8b77-npmzd\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.658924 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.658882 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:04.960810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:04.960776 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd"] Oct 31 21:28:04.961916 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:28:04.961891 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e0bae2f_c86a_487e_ac13_6b7890092979.slice/crio-819cc292d0e660b70f724a5c574c59a8afbac3eee537c0c9d85eb5521b410592 WatchSource:0}: Error finding container 819cc292d0e660b70f724a5c574c59a8afbac3eee537c0c9d85eb5521b410592: Status 404 returned error can't find the container with id 819cc292d0e660b70f724a5c574c59a8afbac3eee537c0c9d85eb5521b410592 Oct 31 21:28:05.310858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:05.310766 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" event={"ID":"9e0bae2f-c86a-487e-ac13-6b7890092979","Type":"ContainerStarted","Data":"e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8"} Oct 31 21:28:05.310858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:05.310806 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" event={"ID":"9e0bae2f-c86a-487e-ac13-6b7890092979","Type":"ContainerStarted","Data":"819cc292d0e660b70f724a5c574c59a8afbac3eee537c0c9d85eb5521b410592"} Oct 31 21:28:05.358896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:05.358831 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podStartSLOduration=1.35881268 podStartE2EDuration="1.35881268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:28:05.354134658 +0000 UTC m=+732.047092050" watchObservedRunningTime="2026-10-31 21:28:05.35881268 +0000 UTC m=+732.051770072" Oct 31 21:28:05.659794 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:05.659757 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:14.659862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:14.659827 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:14.673936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:14.673907 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:15.408655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:15.408626 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:28:32.539866 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:28:32.539825 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:28:42.235952 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.235903 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh"] Oct 31 21:28:42.239821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.239803 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.261727 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.261695 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-106555-openshift-default-dockercfg-v5bxg\"" Oct 31 21:28:42.269042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.269015 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh"] Oct 31 21:28:42.386655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.386613 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.386655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.386654 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c89147-f74d-4df3-b722-48fbf418de11-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.386936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.386828 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c89147-f74d-4df3-b722-48fbf418de11-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.386985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.386942 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.386985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.386972 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.387091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.386995 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.387091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.387068 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.387176 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.387110 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.387209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.387189 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4brql\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-kube-api-access-4brql\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488227 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488274 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488304 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4brql\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-kube-api-access-4brql\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488368 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488385 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c89147-f74d-4df3-b722-48fbf418de11-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488441 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c89147-f74d-4df3-b722-48fbf418de11-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488479 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488496 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488522 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488740 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488840 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488896 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.488989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.488936 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.489317 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.489295 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c89147-f74d-4df3-b722-48fbf418de11-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.490812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.490792 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.490961 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.490943 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c89147-f74d-4df3-b722-48fbf418de11-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.509971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.509939 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.510149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.510028 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4brql\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-kube-api-access-4brql\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.554508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.554467 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:42.877202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.877168 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh"] Oct 31 21:28:42.879111 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:28:42.879080 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c89147_f74d_4df3_b722_48fbf418de11.slice/crio-09ec76dacd6c897e5ce25b866c9838f104b4251cd53ff5e9667bc4f756dff929 WatchSource:0}: Error finding container 09ec76dacd6c897e5ce25b866c9838f104b4251cd53ff5e9667bc4f756dff929: Status 404 returned error can't find the container with id 09ec76dacd6c897e5ce25b866c9838f104b4251cd53ff5e9667bc4f756dff929 Oct 31 21:28:42.881580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.881538 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:28:42.881672 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.881645 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:28:42.881712 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:42.881688 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:28:43.610417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:43.610376 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" event={"ID":"d4c89147-f74d-4df3-b722-48fbf418de11","Type":"ContainerStarted","Data":"f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583"} Oct 31 21:28:43.610857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:43.610449 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" event={"ID":"d4c89147-f74d-4df3-b722-48fbf418de11","Type":"ContainerStarted","Data":"09ec76dacd6c897e5ce25b866c9838f104b4251cd53ff5e9667bc4f756dff929"} Oct 31 21:28:43.672349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:43.671796 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" podStartSLOduration=1.671777337 podStartE2EDuration="1.671777337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:28:43.669966851 +0000 UTC m=+770.362924244" watchObservedRunningTime="2026-10-31 21:28:43.671777337 +0000 UTC m=+770.364734735" Oct 31 21:28:44.555588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:44.555550 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:44.557141 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:44.557108 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 21:28:44.557290 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:44.557187 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 21:28:45.555832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:45.555792 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 21:28:45.556354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:45.555863 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 21:28:46.555421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:46.555378 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 21:28:46.555621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:46.555438 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 21:28:47.555146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:47.555102 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 21:28:47.555565 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:47.555172 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 21:28:48.554854 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:48.554814 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 21:28:48.555036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:48.554880 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 21:28:49.474582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.474544 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb"] Oct 31 21:28:49.478672 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.478646 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.494519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.494477 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:28:49.495258 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.495218 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb"] Oct 31 21:28:49.499349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.499294 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:28:49.499349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.499294 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:28:49.499532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.499401 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:28:49.559587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.559556 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:49.559957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.559935 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:49.560634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.560613 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 21:28:49.578013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.577983 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5605c0d-jfnjb\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.578179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.578018 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5605c0d-jfnjb\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.578373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.578310 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcrbx\" (UniqueName: \"kubernetes.io/projected/86db3058-5a9c-4455-89b6-ff9248670691-kube-api-access-zcrbx\") pod \"maas-api-revoke-sub-f5605c0d-jfnjb\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.681254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.680765 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5605c0d-jfnjb\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.681254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.680987 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcrbx\" (UniqueName: \"kubernetes.io/projected/86db3058-5a9c-4455-89b6-ff9248670691-kube-api-access-zcrbx\") pod \"maas-api-revoke-sub-f5605c0d-jfnjb\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.681254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.681044 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5605c0d-jfnjb\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.681841 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.681818 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5605c0d-jfnjb\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.684524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.684494 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5605c0d-jfnjb\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.703959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.703930 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcrbx\" (UniqueName: \"kubernetes.io/projected/86db3058-5a9c-4455-89b6-ff9248670691-kube-api-access-zcrbx\") pod \"maas-api-revoke-sub-f5605c0d-jfnjb\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:49.798559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:49.798463 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:50.108398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.108366 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb"] Oct 31 21:28:50.108916 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:28:50.108873 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86db3058_5a9c_4455_89b6_ff9248670691.slice/crio-caa357ea895fafeb7c273b18d623dfce7d2fe449ae5b702091e3cf792fde5a61 WatchSource:0}: Error finding container caa357ea895fafeb7c273b18d623dfce7d2fe449ae5b702091e3cf792fde5a61: Status 404 returned error can't find the container with id caa357ea895fafeb7c273b18d623dfce7d2fe449ae5b702091e3cf792fde5a61 Oct 31 21:28:50.126561 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.126527 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh"] Oct 31 21:28:50.131739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.131720 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.151857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.151828 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh"] Oct 31 21:28:50.185390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.185337 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be48c40a-xpbwh\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.185557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.185416 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr2rn\" (UniqueName: \"kubernetes.io/projected/e4352ee1-e585-4981-805c-b58d7215f6d2-kube-api-access-vr2rn\") pod \"maas-api-revoke-sub-be48c40a-xpbwh\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.185557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.185538 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be48c40a-xpbwh\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.286340 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.286286 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be48c40a-xpbwh\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.286537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.286362 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr2rn\" (UniqueName: \"kubernetes.io/projected/e4352ee1-e585-4981-805c-b58d7215f6d2-kube-api-access-vr2rn\") pod \"maas-api-revoke-sub-be48c40a-xpbwh\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.286537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.286464 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be48c40a-xpbwh\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.287231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.287206 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be48c40a-xpbwh\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.288787 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.288764 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be48c40a-xpbwh\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.308263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.308228 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr2rn\" (UniqueName: \"kubernetes.io/projected/e4352ee1-e585-4981-805c-b58d7215f6d2-kube-api-access-vr2rn\") pod \"maas-api-revoke-sub-be48c40a-xpbwh\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.511463 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.511436 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:50.667813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.667716 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" event={"ID":"86db3058-5a9c-4455-89b6-ff9248670691","Type":"ContainerStarted","Data":"41466d4e6b35be3c810b024d6ccd62011b120271be4d5f55afa15e94711f6c69"} Oct 31 21:28:50.667813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.667781 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" event={"ID":"86db3058-5a9c-4455-89b6-ff9248670691","Type":"ContainerStarted","Data":"caa357ea895fafeb7c273b18d623dfce7d2fe449ae5b702091e3cf792fde5a61"} Oct 31 21:28:50.707528 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.707457 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" podStartSLOduration=1.707438776 podStartE2EDuration="1.707438776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:28:50.704656138 +0000 UTC m=+777.397613542" watchObservedRunningTime="2026-10-31 21:28:50.707438776 +0000 UTC m=+777.400396169" Oct 31 21:28:50.809973 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:50.809939 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh"] Oct 31 21:28:50.811396 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:28:50.811316 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4352ee1_e585_4981_805c_b58d7215f6d2.slice/crio-da52f3eeac496d5c50d9f964a5f210c3dfe570f6ac57136e4232d72315136f56 WatchSource:0}: Error finding container da52f3eeac496d5c50d9f964a5f210c3dfe570f6ac57136e4232d72315136f56: Status 404 returned error can't find the container with id da52f3eeac496d5c50d9f964a5f210c3dfe570f6ac57136e4232d72315136f56 Oct 31 21:28:51.676049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:51.676014 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" event={"ID":"e4352ee1-e585-4981-805c-b58d7215f6d2","Type":"ContainerStarted","Data":"276d2094db1437e056d72a6cfe389d13b7cea159194c8f43c1b2e866a6776b46"} Oct 31 21:28:51.676557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:51.676070 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" event={"ID":"e4352ee1-e585-4981-805c-b58d7215f6d2","Type":"ContainerStarted","Data":"da52f3eeac496d5c50d9f964a5f210c3dfe570f6ac57136e4232d72315136f56"} Oct 31 21:28:51.677882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:51.677854 5188 generic.go:356] "Generic (PLEG): container finished" podID="86db3058-5a9c-4455-89b6-ff9248670691" containerID="41466d4e6b35be3c810b024d6ccd62011b120271be4d5f55afa15e94711f6c69" exitCode=0 Oct 31 21:28:51.678020 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:51.677929 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" event={"ID":"86db3058-5a9c-4455-89b6-ff9248670691","Type":"ContainerDied","Data":"41466d4e6b35be3c810b024d6ccd62011b120271be4d5f55afa15e94711f6c69"} Oct 31 21:28:51.713722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:51.713667 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" podStartSLOduration=1.713650329 podStartE2EDuration="1.713650329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:28:51.711702616 +0000 UTC m=+778.404660006" watchObservedRunningTime="2026-10-31 21:28:51.713650329 +0000 UTC m=+778.406607721" Oct 31 21:28:52.687215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.687176 5188 generic.go:356] "Generic (PLEG): container finished" podID="e4352ee1-e585-4981-805c-b58d7215f6d2" containerID="276d2094db1437e056d72a6cfe389d13b7cea159194c8f43c1b2e866a6776b46" exitCode=0 Oct 31 21:28:52.687706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.687264 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" event={"ID":"e4352ee1-e585-4981-805c-b58d7215f6d2","Type":"ContainerDied","Data":"276d2094db1437e056d72a6cfe389d13b7cea159194c8f43c1b2e866a6776b46"} Oct 31 21:28:52.851689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.851656 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:52.914106 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.914064 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86db3058-5a9c-4455-89b6-ff9248670691-kube-api-access-zcrbx\" (UniqueName: \"kubernetes.io/projected/86db3058-5a9c-4455-89b6-ff9248670691-kube-api-access-zcrbx\") pod \"86db3058-5a9c-4455-89b6-ff9248670691\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " Oct 31 21:28:52.914293 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.914276 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-openshift-ca\") pod \"86db3058-5a9c-4455-89b6-ff9248670691\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " Oct 31 21:28:52.914416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.914370 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-xks-ca\") pod \"86db3058-5a9c-4455-89b6-ff9248670691\" (UID: \"86db3058-5a9c-4455-89b6-ff9248670691\") " Oct 31 21:28:52.914717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.914685 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-openshift-ca" pod "86db3058-5a9c-4455-89b6-ff9248670691" (UID: "86db3058-5a9c-4455-89b6-ff9248670691"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:28:52.914810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.914790 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:28:52.916521 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.916488 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86db3058-5a9c-4455-89b6-ff9248670691-kube-api-access-zcrbx" pod "86db3058-5a9c-4455-89b6-ff9248670691" (UID: "86db3058-5a9c-4455-89b6-ff9248670691"). InnerVolumeSpecName "kube-api-access-zcrbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:28:52.916779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:52.916756 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-xks-ca" pod "86db3058-5a9c-4455-89b6-ff9248670691" (UID: "86db3058-5a9c-4455-89b6-ff9248670691"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:28:53.015516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.015431 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86db3058-5a9c-4455-89b6-ff9248670691-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:28:53.015516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.015457 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcrbx\" (UniqueName: \"kubernetes.io/projected/86db3058-5a9c-4455-89b6-ff9248670691-kube-api-access-zcrbx\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:28:53.694295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.694270 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" Oct 31 21:28:53.694670 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.694305 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb" event={"ID":"86db3058-5a9c-4455-89b6-ff9248670691","Type":"ContainerDied","Data":"caa357ea895fafeb7c273b18d623dfce7d2fe449ae5b702091e3cf792fde5a61"} Oct 31 21:28:53.694670 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.694369 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caa357ea895fafeb7c273b18d623dfce7d2fe449ae5b702091e3cf792fde5a61" Oct 31 21:28:53.852787 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.852761 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:53.925258 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.925224 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-openshift-ca\") pod \"e4352ee1-e585-4981-805c-b58d7215f6d2\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " Oct 31 21:28:53.925457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.925286 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4352ee1-e585-4981-805c-b58d7215f6d2-kube-api-access-vr2rn\" (UniqueName: \"kubernetes.io/projected/e4352ee1-e585-4981-805c-b58d7215f6d2-kube-api-access-vr2rn\") pod \"e4352ee1-e585-4981-805c-b58d7215f6d2\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " Oct 31 21:28:53.925545 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.925466 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-xks-ca\") pod \"e4352ee1-e585-4981-805c-b58d7215f6d2\" (UID: \"e4352ee1-e585-4981-805c-b58d7215f6d2\") " Oct 31 21:28:53.925663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.925634 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-openshift-ca" pod "e4352ee1-e585-4981-805c-b58d7215f6d2" (UID: "e4352ee1-e585-4981-805c-b58d7215f6d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:28:53.925966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.925912 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:28:53.927488 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.927467 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4352ee1-e585-4981-805c-b58d7215f6d2-kube-api-access-vr2rn" pod "e4352ee1-e585-4981-805c-b58d7215f6d2" (UID: "e4352ee1-e585-4981-805c-b58d7215f6d2"). InnerVolumeSpecName "kube-api-access-vr2rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:28:53.927698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:53.927684 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-xks-ca" pod "e4352ee1-e585-4981-805c-b58d7215f6d2" (UID: "e4352ee1-e585-4981-805c-b58d7215f6d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:28:54.027229 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:54.027152 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4352ee1-e585-4981-805c-b58d7215f6d2-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:28:54.027229 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:54.027178 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr2rn\" (UniqueName: \"kubernetes.io/projected/e4352ee1-e585-4981-805c-b58d7215f6d2-kube-api-access-vr2rn\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:28:54.702318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:54.702284 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" event={"ID":"e4352ee1-e585-4981-805c-b58d7215f6d2","Type":"ContainerDied","Data":"da52f3eeac496d5c50d9f964a5f210c3dfe570f6ac57136e4232d72315136f56"} Oct 31 21:28:54.702318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:54.702296 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh" Oct 31 21:28:54.702773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:54.702355 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da52f3eeac496d5c50d9f964a5f210c3dfe570f6ac57136e4232d72315136f56" Oct 31 21:28:58.619267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.619229 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz"] Oct 31 21:28:58.619790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.619774 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e4352ee1-e585-4981-805c-b58d7215f6d2" containerName="revoke-keys" Oct 31 21:28:58.619852 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.619793 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4352ee1-e585-4981-805c-b58d7215f6d2" containerName="revoke-keys" Oct 31 21:28:58.619852 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.619845 5188 cpu_manager.go:414] "containerMap: removing container" podUID="86db3058-5a9c-4455-89b6-ff9248670691" containerName="revoke-keys" Oct 31 21:28:58.619916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.619853 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86db3058-5a9c-4455-89b6-ff9248670691" containerName="revoke-keys" Oct 31 21:28:58.619972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.619960 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="86db3058-5a9c-4455-89b6-ff9248670691" containerName="revoke-keys" Oct 31 21:28:58.620006 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.619974 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4352ee1-e585-4981-805c-b58d7215f6d2" containerName="revoke-keys" Oct 31 21:28:58.623817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.623796 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.636302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.636275 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:28:58.647554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.647524 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:28:58.647780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.647595 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:28:58.647780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.647713 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:28:58.653037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.653006 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz"] Oct 31 21:28:58.676466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.676429 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttp9\" (UniqueName: \"kubernetes.io/projected/cb874929-a459-40a7-99a4-31769e515dcd-kube-api-access-pttp9\") pod \"maas-api-revoke-sub-6ea69b3c-h8fsz\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.676667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.676535 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ea69b3c-h8fsz\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.676747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.676710 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ea69b3c-h8fsz\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.778091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.778054 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttp9\" (UniqueName: \"kubernetes.io/projected/cb874929-a459-40a7-99a4-31769e515dcd-kube-api-access-pttp9\") pod \"maas-api-revoke-sub-6ea69b3c-h8fsz\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.778307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.778099 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ea69b3c-h8fsz\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.778307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.778195 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ea69b3c-h8fsz\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.778885 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.778857 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ea69b3c-h8fsz\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.780656 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.780631 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ea69b3c-h8fsz\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.799379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.799349 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttp9\" (UniqueName: \"kubernetes.io/projected/cb874929-a459-40a7-99a4-31769e515dcd-kube-api-access-pttp9\") pod \"maas-api-revoke-sub-6ea69b3c-h8fsz\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:58.939297 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:58.939214 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:28:59.246453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:59.246423 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz"] Oct 31 21:28:59.247661 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:28:59.247633 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb874929_a459_40a7_99a4_31769e515dcd.slice/crio-89927a6f8acd07338e7a81e91cf67ab7ed1ee533f2854ed7681c655a8848defe WatchSource:0}: Error finding container 89927a6f8acd07338e7a81e91cf67ab7ed1ee533f2854ed7681c655a8848defe: Status 404 returned error can't find the container with id 89927a6f8acd07338e7a81e91cf67ab7ed1ee533f2854ed7681c655a8848defe Oct 31 21:28:59.749303 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:59.749262 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" event={"ID":"cb874929-a459-40a7-99a4-31769e515dcd","Type":"ContainerStarted","Data":"d1b69b1cdda1766bc7b7adbf07f7754b360855d6b5cd340f2f7da042834bed3f"} Oct 31 21:28:59.749819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:59.749332 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" event={"ID":"cb874929-a459-40a7-99a4-31769e515dcd","Type":"ContainerStarted","Data":"89927a6f8acd07338e7a81e91cf67ab7ed1ee533f2854ed7681c655a8848defe"} Oct 31 21:28:59.787999 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:28:59.787933 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" podStartSLOduration=1.787913758 podStartE2EDuration="1.787913758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:28:59.784824087 +0000 UTC m=+786.477781479" watchObservedRunningTime="2026-10-31 21:28:59.787913758 +0000 UTC m=+786.480871150" Oct 31 21:29:00.758300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:00.758259 5188 generic.go:356] "Generic (PLEG): container finished" podID="cb874929-a459-40a7-99a4-31769e515dcd" containerID="d1b69b1cdda1766bc7b7adbf07f7754b360855d6b5cd340f2f7da042834bed3f" exitCode=0 Oct 31 21:29:00.758684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:00.758360 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" event={"ID":"cb874929-a459-40a7-99a4-31769e515dcd","Type":"ContainerDied","Data":"d1b69b1cdda1766bc7b7adbf07f7754b360855d6b5cd340f2f7da042834bed3f"} Oct 31 21:29:01.922712 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:01.922681 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:29:02.017608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.017568 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-openshift-ca\") pod \"cb874929-a459-40a7-99a4-31769e515dcd\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " Oct 31 21:29:02.017783 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.017696 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb874929-a459-40a7-99a4-31769e515dcd-kube-api-access-pttp9\" (UniqueName: \"kubernetes.io/projected/cb874929-a459-40a7-99a4-31769e515dcd-kube-api-access-pttp9\") pod \"cb874929-a459-40a7-99a4-31769e515dcd\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " Oct 31 21:29:02.017783 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.017755 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-xks-ca\") pod \"cb874929-a459-40a7-99a4-31769e515dcd\" (UID: \"cb874929-a459-40a7-99a4-31769e515dcd\") " Oct 31 21:29:02.018045 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.018017 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-openshift-ca" pod "cb874929-a459-40a7-99a4-31769e515dcd" (UID: "cb874929-a459-40a7-99a4-31769e515dcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:29:02.018270 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.018251 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:02.020000 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.019941 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-xks-ca" pod "cb874929-a459-40a7-99a4-31769e515dcd" (UID: "cb874929-a459-40a7-99a4-31769e515dcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:29:02.020000 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.019960 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb874929-a459-40a7-99a4-31769e515dcd-kube-api-access-pttp9" pod "cb874929-a459-40a7-99a4-31769e515dcd" (UID: "cb874929-a459-40a7-99a4-31769e515dcd"). InnerVolumeSpecName "kube-api-access-pttp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:29:02.119056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.119018 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttp9\" (UniqueName: \"kubernetes.io/projected/cb874929-a459-40a7-99a4-31769e515dcd-kube-api-access-pttp9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:02.119056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.119050 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb874929-a459-40a7-99a4-31769e515dcd-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:02.537397 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:29:02.537355 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:29:02.777015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.776989 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" Oct 31 21:29:02.777247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.776987 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz" event={"ID":"cb874929-a459-40a7-99a4-31769e515dcd","Type":"ContainerDied","Data":"89927a6f8acd07338e7a81e91cf67ab7ed1ee533f2854ed7681c655a8848defe"} Oct 31 21:29:02.777247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:02.777117 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89927a6f8acd07338e7a81e91cf67ab7ed1ee533f2854ed7681c655a8848defe" Oct 31 21:29:04.512845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.512811 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz"] Oct 31 21:29:04.513348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.513333 5188 cpu_manager.go:414] "containerMap: removing container" podUID="cb874929-a459-40a7-99a4-31769e515dcd" containerName="revoke-keys" Oct 31 21:29:04.513392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.513348 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb874929-a459-40a7-99a4-31769e515dcd" containerName="revoke-keys" Oct 31 21:29:04.513470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.513460 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb874929-a459-40a7-99a4-31769e515dcd" containerName="revoke-keys" Oct 31 21:29:04.517104 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.517074 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.529712 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.529683 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:29:04.539429 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.539396 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:29:04.539643 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.539509 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:29:04.539643 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.539538 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:29:04.543024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.542998 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz"] Oct 31 21:29:04.646946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.646905 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32af2159-ppsbz\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.647149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.646978 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7v7\" (UniqueName: \"kubernetes.io/projected/b105f6f6-4640-4e02-8edb-45f9cd86f092-kube-api-access-9q7v7\") pod \"maas-api-revoke-sub-32af2159-ppsbz\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.647149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.647063 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32af2159-ppsbz\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.748354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.748280 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7v7\" (UniqueName: \"kubernetes.io/projected/b105f6f6-4640-4e02-8edb-45f9cd86f092-kube-api-access-9q7v7\") pod \"maas-api-revoke-sub-32af2159-ppsbz\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.748354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.748355 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32af2159-ppsbz\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.748576 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.748507 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32af2159-ppsbz\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.749386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.749311 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32af2159-ppsbz\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.750362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.750316 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt"] Oct 31 21:29:04.751002 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.750981 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32af2159-ppsbz\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.754423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.754402 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:04.770399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.770358 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7v7\" (UniqueName: \"kubernetes.io/projected/b105f6f6-4640-4e02-8edb-45f9cd86f092-kube-api-access-9q7v7\") pod \"maas-api-revoke-sub-32af2159-ppsbz\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.775663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.775628 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt"] Oct 31 21:29:04.833241 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.833202 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:04.850099 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.850059 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65eb6c27-xxwlt\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:04.850352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.850300 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zlp\" (UniqueName: \"kubernetes.io/projected/5e00c871-5272-4032-90b1-00a6e69c9909-kube-api-access-49zlp\") pod \"maas-api-revoke-sub-65eb6c27-xxwlt\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:04.850467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.850449 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65eb6c27-xxwlt\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:04.952207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.952158 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zlp\" (UniqueName: \"kubernetes.io/projected/5e00c871-5272-4032-90b1-00a6e69c9909-kube-api-access-49zlp\") pod \"maas-api-revoke-sub-65eb6c27-xxwlt\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:04.952437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.952245 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65eb6c27-xxwlt\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:04.952437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.952347 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65eb6c27-xxwlt\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:04.953091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.953067 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65eb6c27-xxwlt\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:04.954901 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.954873 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65eb6c27-xxwlt\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:04.973788 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:04.973753 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zlp\" (UniqueName: \"kubernetes.io/projected/5e00c871-5272-4032-90b1-00a6e69c9909-kube-api-access-49zlp\") pod \"maas-api-revoke-sub-65eb6c27-xxwlt\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:05.078552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:05.078463 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:05.163541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:05.158944 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz"] Oct 31 21:29:05.184974 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:29:05.184936 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb105f6f6_4640_4e02_8edb_45f9cd86f092.slice/crio-b4e0c3ed1668b803eb63ec998c01887d51e253c2b934dd9c85dbebfd531aca58 WatchSource:0}: Error finding container b4e0c3ed1668b803eb63ec998c01887d51e253c2b934dd9c85dbebfd531aca58: Status 404 returned error can't find the container with id b4e0c3ed1668b803eb63ec998c01887d51e253c2b934dd9c85dbebfd531aca58 Oct 31 21:29:05.424679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:05.424644 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt"] Oct 31 21:29:05.427784 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:29:05.427741 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e00c871_5272_4032_90b1_00a6e69c9909.slice/crio-1b670a32694945c86c43cea4280fcdf2f50c3a5f2faf4ae25fd174f46b6d81f8 WatchSource:0}: Error finding container 1b670a32694945c86c43cea4280fcdf2f50c3a5f2faf4ae25fd174f46b6d81f8: Status 404 returned error can't find the container with id 1b670a32694945c86c43cea4280fcdf2f50c3a5f2faf4ae25fd174f46b6d81f8 Oct 31 21:29:05.815376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:05.808201 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" event={"ID":"b105f6f6-4640-4e02-8edb-45f9cd86f092","Type":"ContainerStarted","Data":"599a5fe0a34f9a95e678e0b8a1a4c2bf72dba961a25c3f7bd4d7626623ca8afd"} Oct 31 21:29:05.815376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:05.808273 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" event={"ID":"b105f6f6-4640-4e02-8edb-45f9cd86f092","Type":"ContainerStarted","Data":"b4e0c3ed1668b803eb63ec998c01887d51e253c2b934dd9c85dbebfd531aca58"} Oct 31 21:29:05.815376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:05.810964 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" event={"ID":"5e00c871-5272-4032-90b1-00a6e69c9909","Type":"ContainerStarted","Data":"55238865cca5a67f8a82b6fc028b58a48dce586d6f56e4ab6dea73c0bae12518"} Oct 31 21:29:05.815376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:05.811023 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" event={"ID":"5e00c871-5272-4032-90b1-00a6e69c9909","Type":"ContainerStarted","Data":"1b670a32694945c86c43cea4280fcdf2f50c3a5f2faf4ae25fd174f46b6d81f8"} Oct 31 21:29:05.847562 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:05.847500 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" podStartSLOduration=1.847485427 podStartE2EDuration="1.847485427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:05.845850914 +0000 UTC m=+792.538808304" watchObservedRunningTime="2026-10-31 21:29:05.847485427 +0000 UTC m=+792.540442818" Oct 31 21:29:05.884497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:05.884401 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" podStartSLOduration=1.884388199 podStartE2EDuration="1.884388199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:05.881112473 +0000 UTC m=+792.574069865" watchObservedRunningTime="2026-10-31 21:29:05.884388199 +0000 UTC m=+792.577345589" Oct 31 21:29:06.823163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:06.823064 5188 generic.go:356] "Generic (PLEG): container finished" podID="b105f6f6-4640-4e02-8edb-45f9cd86f092" containerID="599a5fe0a34f9a95e678e0b8a1a4c2bf72dba961a25c3f7bd4d7626623ca8afd" exitCode=0 Oct 31 21:29:06.823163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:06.823115 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" event={"ID":"b105f6f6-4640-4e02-8edb-45f9cd86f092","Type":"ContainerDied","Data":"599a5fe0a34f9a95e678e0b8a1a4c2bf72dba961a25c3f7bd4d7626623ca8afd"} Oct 31 21:29:06.824839 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:06.824816 5188 generic.go:356] "Generic (PLEG): container finished" podID="5e00c871-5272-4032-90b1-00a6e69c9909" containerID="55238865cca5a67f8a82b6fc028b58a48dce586d6f56e4ab6dea73c0bae12518" exitCode=0 Oct 31 21:29:06.824960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:06.824852 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" event={"ID":"5e00c871-5272-4032-90b1-00a6e69c9909","Type":"ContainerDied","Data":"55238865cca5a67f8a82b6fc028b58a48dce586d6f56e4ab6dea73c0bae12518"} Oct 31 21:29:08.031142 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.031106 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:08.035162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.035137 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:08.093138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.093047 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-openshift-ca\") pod \"5e00c871-5272-4032-90b1-00a6e69c9909\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " Oct 31 21:29:08.093138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.093098 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-xks-ca\") pod \"5e00c871-5272-4032-90b1-00a6e69c9909\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " Oct 31 21:29:08.093399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.093194 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e00c871-5272-4032-90b1-00a6e69c9909-kube-api-access-49zlp\" (UniqueName: \"kubernetes.io/projected/5e00c871-5272-4032-90b1-00a6e69c9909-kube-api-access-49zlp\") pod \"5e00c871-5272-4032-90b1-00a6e69c9909\" (UID: \"5e00c871-5272-4032-90b1-00a6e69c9909\") " Oct 31 21:29:08.093458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.093430 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-openshift-ca" pod "5e00c871-5272-4032-90b1-00a6e69c9909" (UID: "5e00c871-5272-4032-90b1-00a6e69c9909"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:29:08.093703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.093679 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:08.095951 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.095420 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-xks-ca" pod "5e00c871-5272-4032-90b1-00a6e69c9909" (UID: "5e00c871-5272-4032-90b1-00a6e69c9909"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:29:08.101735 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.101700 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e00c871-5272-4032-90b1-00a6e69c9909-kube-api-access-49zlp" pod "5e00c871-5272-4032-90b1-00a6e69c9909" (UID: "5e00c871-5272-4032-90b1-00a6e69c9909"). InnerVolumeSpecName "kube-api-access-49zlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:29:08.131557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.131513 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn"] Oct 31 21:29:08.132380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.132318 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b105f6f6-4640-4e02-8edb-45f9cd86f092" containerName="revoke-keys" Oct 31 21:29:08.132380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.132381 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b105f6f6-4640-4e02-8edb-45f9cd86f092" containerName="revoke-keys" Oct 31 21:29:08.132634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.132454 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5e00c871-5272-4032-90b1-00a6e69c9909" containerName="revoke-keys" Oct 31 21:29:08.132634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.132463 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e00c871-5272-4032-90b1-00a6e69c9909" containerName="revoke-keys" Oct 31 21:29:08.132634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.132596 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e00c871-5272-4032-90b1-00a6e69c9909" containerName="revoke-keys" Oct 31 21:29:08.132634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.132613 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b105f6f6-4640-4e02-8edb-45f9cd86f092" containerName="revoke-keys" Oct 31 21:29:08.136688 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.136661 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.156289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.156253 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn"] Oct 31 21:29:08.194355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.194303 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b105f6f6-4640-4e02-8edb-45f9cd86f092-kube-api-access-9q7v7\" (UniqueName: \"kubernetes.io/projected/b105f6f6-4640-4e02-8edb-45f9cd86f092-kube-api-access-9q7v7\") pod \"b105f6f6-4640-4e02-8edb-45f9cd86f092\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " Oct 31 21:29:08.194530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.194448 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-xks-ca\") pod \"b105f6f6-4640-4e02-8edb-45f9cd86f092\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " Oct 31 21:29:08.194648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.194631 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-openshift-ca\") pod \"b105f6f6-4640-4e02-8edb-45f9cd86f092\" (UID: \"b105f6f6-4640-4e02-8edb-45f9cd86f092\") " Oct 31 21:29:08.195012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.194963 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-openshift-ca" pod "b105f6f6-4640-4e02-8edb-45f9cd86f092" (UID: "b105f6f6-4640-4e02-8edb-45f9cd86f092"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:29:08.195153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.195068 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e00c871-5272-4032-90b1-00a6e69c9909-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:08.195153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.195090 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49zlp\" (UniqueName: \"kubernetes.io/projected/5e00c871-5272-4032-90b1-00a6e69c9909-kube-api-access-49zlp\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:08.195153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.195106 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:08.196477 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.196452 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-xks-ca" pod "b105f6f6-4640-4e02-8edb-45f9cd86f092" (UID: "b105f6f6-4640-4e02-8edb-45f9cd86f092"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:29:08.196799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.196773 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b105f6f6-4640-4e02-8edb-45f9cd86f092-kube-api-access-9q7v7" pod "b105f6f6-4640-4e02-8edb-45f9cd86f092" (UID: "b105f6f6-4640-4e02-8edb-45f9cd86f092"). InnerVolumeSpecName "kube-api-access-9q7v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:29:08.296584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.296547 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca46659-sjtrn\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.296800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.296748 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4klch\" (UniqueName: \"kubernetes.io/projected/3f9e1334-4113-43bd-b348-5fd17bc24f6a-kube-api-access-4klch\") pod \"maas-api-revoke-sub-bca46659-sjtrn\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.296870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.296823 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca46659-sjtrn\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.296972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.296957 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b105f6f6-4640-4e02-8edb-45f9cd86f092-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:08.297019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.296978 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q7v7\" (UniqueName: \"kubernetes.io/projected/b105f6f6-4640-4e02-8edb-45f9cd86f092-kube-api-access-9q7v7\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:08.398338 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.398281 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca46659-sjtrn\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.398587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.398398 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca46659-sjtrn\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.398587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.398523 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4klch\" (UniqueName: \"kubernetes.io/projected/3f9e1334-4113-43bd-b348-5fd17bc24f6a-kube-api-access-4klch\") pod \"maas-api-revoke-sub-bca46659-sjtrn\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.399124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.399099 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca46659-sjtrn\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.400846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.400795 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca46659-sjtrn\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.429149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.429118 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4klch\" (UniqueName: \"kubernetes.io/projected/3f9e1334-4113-43bd-b348-5fd17bc24f6a-kube-api-access-4klch\") pod \"maas-api-revoke-sub-bca46659-sjtrn\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.451153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.451117 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:08.758698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.758662 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn"] Oct 31 21:29:08.759585 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:29:08.759553 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9e1334_4113_43bd_b348_5fd17bc24f6a.slice/crio-7a7a424567ef960ab702f7fcf191eaf0a6b72b421db3ad3714f6e6792c9a9982 WatchSource:0}: Error finding container 7a7a424567ef960ab702f7fcf191eaf0a6b72b421db3ad3714f6e6792c9a9982: Status 404 returned error can't find the container with id 7a7a424567ef960ab702f7fcf191eaf0a6b72b421db3ad3714f6e6792c9a9982 Oct 31 21:29:08.839139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.839112 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" Oct 31 21:29:08.839348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.839110 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz" event={"ID":"b105f6f6-4640-4e02-8edb-45f9cd86f092","Type":"ContainerDied","Data":"b4e0c3ed1668b803eb63ec998c01887d51e253c2b934dd9c85dbebfd531aca58"} Oct 31 21:29:08.839348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.839238 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4e0c3ed1668b803eb63ec998c01887d51e253c2b934dd9c85dbebfd531aca58" Oct 31 21:29:08.840592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.840560 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" event={"ID":"3f9e1334-4113-43bd-b348-5fd17bc24f6a","Type":"ContainerStarted","Data":"7a7a424567ef960ab702f7fcf191eaf0a6b72b421db3ad3714f6e6792c9a9982"} Oct 31 21:29:08.841953 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.841933 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" Oct 31 21:29:08.842061 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.841960 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt" event={"ID":"5e00c871-5272-4032-90b1-00a6e69c9909","Type":"ContainerDied","Data":"1b670a32694945c86c43cea4280fcdf2f50c3a5f2faf4ae25fd174f46b6d81f8"} Oct 31 21:29:08.842061 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:08.841997 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b670a32694945c86c43cea4280fcdf2f50c3a5f2faf4ae25fd174f46b6d81f8" Oct 31 21:29:09.850433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:09.850398 5188 generic.go:356] "Generic (PLEG): container finished" podID="3f9e1334-4113-43bd-b348-5fd17bc24f6a" containerID="3d03f589aca066d0055f0a1b59cb980bc9f6ec7dc038480ab83469ca917eeda6" exitCode=0 Oct 31 21:29:09.850819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:09.850480 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" event={"ID":"3f9e1334-4113-43bd-b348-5fd17bc24f6a","Type":"ContainerDied","Data":"3d03f589aca066d0055f0a1b59cb980bc9f6ec7dc038480ab83469ca917eeda6"} Oct 31 21:29:11.014726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.014693 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:11.128983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.128944 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9e1334-4113-43bd-b348-5fd17bc24f6a-kube-api-access-4klch\" (UniqueName: \"kubernetes.io/projected/3f9e1334-4113-43bd-b348-5fd17bc24f6a-kube-api-access-4klch\") pod \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " Oct 31 21:29:11.129192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.129038 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-xks-ca\") pod \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " Oct 31 21:29:11.129192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.129134 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-openshift-ca\") pod \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\" (UID: \"3f9e1334-4113-43bd-b348-5fd17bc24f6a\") " Oct 31 21:29:11.129641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.129606 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-openshift-ca" pod "3f9e1334-4113-43bd-b348-5fd17bc24f6a" (UID: "3f9e1334-4113-43bd-b348-5fd17bc24f6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:29:11.131127 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.131100 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-xks-ca" pod "3f9e1334-4113-43bd-b348-5fd17bc24f6a" (UID: "3f9e1334-4113-43bd-b348-5fd17bc24f6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:29:11.131240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.131207 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9e1334-4113-43bd-b348-5fd17bc24f6a-kube-api-access-4klch" pod "3f9e1334-4113-43bd-b348-5fd17bc24f6a" (UID: "3f9e1334-4113-43bd-b348-5fd17bc24f6a"). InnerVolumeSpecName "kube-api-access-4klch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:29:11.230760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.230719 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:11.230760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.230752 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9e1334-4113-43bd-b348-5fd17bc24f6a-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:11.230760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.230763 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4klch\" (UniqueName: \"kubernetes.io/projected/3f9e1334-4113-43bd-b348-5fd17bc24f6a-kube-api-access-4klch\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:11.865141 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.865109 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" Oct 31 21:29:11.865346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.865107 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn" event={"ID":"3f9e1334-4113-43bd-b348-5fd17bc24f6a","Type":"ContainerDied","Data":"7a7a424567ef960ab702f7fcf191eaf0a6b72b421db3ad3714f6e6792c9a9982"} Oct 31 21:29:11.865346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:11.865236 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a7a424567ef960ab702f7fcf191eaf0a6b72b421db3ad3714f6e6792c9a9982" Oct 31 21:29:13.569797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.569755 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2"] Oct 31 21:29:13.570517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.570499 5188 cpu_manager.go:414] "containerMap: removing container" podUID="3f9e1334-4113-43bd-b348-5fd17bc24f6a" containerName="revoke-keys" Oct 31 21:29:13.570604 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.570517 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9e1334-4113-43bd-b348-5fd17bc24f6a" containerName="revoke-keys" Oct 31 21:29:13.570693 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.570682 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9e1334-4113-43bd-b348-5fd17bc24f6a" containerName="revoke-keys" Oct 31 21:29:13.574629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.574600 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.603790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.603755 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4b5989-openshift-default-dockercfg-gpzdz\"" Oct 31 21:29:13.607646 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.607612 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2"] Oct 31 21:29:13.655780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.655736 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqskw\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-kube-api-access-sqskw\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.655780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.655781 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776d7137-587e-4c9a-8fbb-648a39b77e34-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.656005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.655876 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.656005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.655940 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.656117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.656020 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.656117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.656081 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.656204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.656140 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.656334 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.656304 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.656415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.656357 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.757687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.757650 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.757874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.757701 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.757874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.757745 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqskw\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-kube-api-access-sqskw\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.757874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.757769 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776d7137-587e-4c9a-8fbb-648a39b77e34-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.757874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.757812 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.758094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.757875 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.758094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.757914 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.758094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.757943 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.758094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.758025 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.758283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.758245 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.758592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.758568 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.758659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.758603 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.758659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.758632 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.758738 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.758717 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776d7137-587e-4c9a-8fbb-648a39b77e34-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.760609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.760583 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.761045 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.760995 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.784443 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.784405 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqskw\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-kube-api-access-sqskw\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.786407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.786381 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:13.892550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:13.892505 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:14.216282 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.216248 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2"] Oct 31 21:29:14.217573 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:29:14.217546 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776d7137_587e_4c9a_8fbb_648a39b77e34.slice/crio-214fde7249b6640f4512fc928a2d2a8803bfb7e3ae6fcc6a7b173e860cb275fa WatchSource:0}: Error finding container 214fde7249b6640f4512fc928a2d2a8803bfb7e3ae6fcc6a7b173e860cb275fa: Status 404 returned error can't find the container with id 214fde7249b6640f4512fc928a2d2a8803bfb7e3ae6fcc6a7b173e860cb275fa Oct 31 21:29:14.220415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.220373 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:29:14.220529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.220464 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:29:14.220529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.220514 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:29:14.625549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.625515 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg"] Oct 31 21:29:14.629547 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.629525 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.643265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.643215 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 21:29:14.653351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.652209 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 21:29:14.653519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.653500 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 21:29:14.653595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.653580 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 21:29:14.653800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.653773 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-975bec2bdac9\"" Oct 31 21:29:14.666663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.655651 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0c70771b6656\"" Oct 31 21:29:14.666663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.655840 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mbzjk\"" Oct 31 21:29:14.666663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.662555 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg"] Oct 31 21:29:14.775346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.771669 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-plugins-config-volume\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.775346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.771723 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-routing-overlay-volume\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.775346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.771753 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-provider-credentials\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.775346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.771931 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6vmb\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-kube-api-access-l6vmb\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.879393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.873406 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-plugins-config-volume\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.879393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.873463 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-routing-overlay-volume\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.879393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.873491 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-provider-credentials\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.879393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.873661 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6vmb\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-kube-api-access-l6vmb\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.879393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.874942 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-routing-overlay-volume\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.879393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.875866 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-plugins-config-volume\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.879393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.878645 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-provider-credentials\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.903833 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.903793 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" event={"ID":"776d7137-587e-4c9a-8fbb-648a39b77e34","Type":"ContainerStarted","Data":"bbecb38fc37676bc8a76ec2d348d488b31701bdf5148de4994600bc9f376ac65"} Oct 31 21:29:14.904068 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.904046 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" event={"ID":"776d7137-587e-4c9a-8fbb-648a39b77e34","Type":"ContainerStarted","Data":"214fde7249b6640f4512fc928a2d2a8803bfb7e3ae6fcc6a7b173e860cb275fa"} Oct 31 21:29:14.905492 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.905459 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6vmb\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-kube-api-access-l6vmb\") pod \"payload-processing-external-model-78fb45f9df-z72zg\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.947793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.947746 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:14.973609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:14.969868 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" podStartSLOduration=1.969845448 podStartE2EDuration="1.969845448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:14.951375838 +0000 UTC m=+801.644333255" watchObservedRunningTime="2026-10-31 21:29:14.969845448 +0000 UTC m=+801.662802841" Oct 31 21:29:15.883642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:15.883592 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg"] Oct 31 21:29:15.893793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:15.893746 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:15.896786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:15.896750 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 21:29:15.897063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:15.897029 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 21:29:15.945001 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:15.944910 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" event={"ID":"dba98a77-478c-494f-8026-3a67d2ad039c","Type":"ContainerStarted","Data":"1c874d4fbed669e2cd79d72a0421ed3f3bfdb96829aadb2826a324ffd6e9b02f"} Oct 31 21:29:16.895355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:16.894759 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 21:29:16.895355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:16.894850 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 21:29:16.999980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:16.999944 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" event={"ID":"dba98a77-478c-494f-8026-3a67d2ad039c","Type":"ContainerStarted","Data":"cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16"} Oct 31 21:29:17.000304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:17.000280 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:17.057353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:17.049317 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" podStartSLOduration=3.049298448 podStartE2EDuration="3.049298448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:17.048758278 +0000 UTC m=+803.741715670" watchObservedRunningTime="2026-10-31 21:29:17.049298448 +0000 UTC m=+803.742255843" Oct 31 21:29:17.564349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:17.557499 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:29:17.564349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:17.557575 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:29:17.897031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:17.896980 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 21:29:17.897715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:17.897661 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 21:29:18.692594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.652056 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl"] Oct 31 21:29:18.749581 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.749544 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl"] Oct 31 21:29:18.749781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.749681 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:18.767371 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.767339 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:29:18.767910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.767879 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:29:18.777124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.777092 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:29:18.777479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.777458 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:29:18.837241 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.837197 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4ad3fa-qtddl\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:18.837483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.837364 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4ad3fa-qtddl\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:18.837483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.837414 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpt2\" (UniqueName: \"kubernetes.io/projected/7866c78e-aaae-4fdb-948e-9facecee1434-kube-api-access-bfpt2\") pod \"maas-api-revoke-sub-8f4ad3fa-qtddl\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:18.895871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.895827 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 21:29:18.896538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.896503 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 21:29:18.938582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.938536 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4ad3fa-qtddl\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:18.939017 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.938601 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4ad3fa-qtddl\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:18.939017 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.938626 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpt2\" (UniqueName: \"kubernetes.io/projected/7866c78e-aaae-4fdb-948e-9facecee1434-kube-api-access-bfpt2\") pod \"maas-api-revoke-sub-8f4ad3fa-qtddl\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:18.939230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.939207 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f4ad3fa-qtddl\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:18.941051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.941028 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f4ad3fa-qtddl\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:18.962741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:18.962670 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpt2\" (UniqueName: \"kubernetes.io/projected/7866c78e-aaae-4fdb-948e-9facecee1434-kube-api-access-bfpt2\") pod \"maas-api-revoke-sub-8f4ad3fa-qtddl\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:19.069703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.069668 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:19.262872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.262792 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb"] Oct 31 21:29:19.268057 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.268035 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.312511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.312373 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb"] Oct 31 21:29:19.343905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.343856 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ebf5804-ndcdb\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.344107 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.344009 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxbn\" (UniqueName: \"kubernetes.io/projected/8bde2fde-9142-40c6-bebc-22e38137ae60-kube-api-access-dpxbn\") pod \"maas-api-revoke-sub-6ebf5804-ndcdb\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.344107 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.344071 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ebf5804-ndcdb\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.409725 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.409693 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl"] Oct 31 21:29:19.410971 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:29:19.410938 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7866c78e_aaae_4fdb_948e_9facecee1434.slice/crio-468c37880594ac27611a80312d35ff566a4cddb5bfe8fddce572f80a283b4f59 WatchSource:0}: Error finding container 468c37880594ac27611a80312d35ff566a4cddb5bfe8fddce572f80a283b4f59: Status 404 returned error can't find the container with id 468c37880594ac27611a80312d35ff566a4cddb5bfe8fddce572f80a283b4f59 Oct 31 21:29:19.445620 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.445589 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ebf5804-ndcdb\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.445760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.445703 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ebf5804-ndcdb\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.445760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.445753 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxbn\" (UniqueName: \"kubernetes.io/projected/8bde2fde-9142-40c6-bebc-22e38137ae60-kube-api-access-dpxbn\") pod \"maas-api-revoke-sub-6ebf5804-ndcdb\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.446260 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.446238 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ebf5804-ndcdb\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.448063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.448040 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ebf5804-ndcdb\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.467829 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.467793 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxbn\" (UniqueName: \"kubernetes.io/projected/8bde2fde-9142-40c6-bebc-22e38137ae60-kube-api-access-dpxbn\") pod \"maas-api-revoke-sub-6ebf5804-ndcdb\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.623382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.623300 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:19.898625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.898510 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 21:29:19.898625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.898583 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 21:29:19.966911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:19.966874 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb"] Oct 31 21:29:19.967692 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:29:19.967665 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bde2fde_9142_40c6_bebc_22e38137ae60.slice/crio-8f4805577180f6203ff1ebc92c8dac2d2ca79f35615d5b6f10c0fe28abe80bbe WatchSource:0}: Error finding container 8f4805577180f6203ff1ebc92c8dac2d2ca79f35615d5b6f10c0fe28abe80bbe: Status 404 returned error can't find the container with id 8f4805577180f6203ff1ebc92c8dac2d2ca79f35615d5b6f10c0fe28abe80bbe Oct 31 21:29:20.046920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:20.046877 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" event={"ID":"7866c78e-aaae-4fdb-948e-9facecee1434","Type":"ContainerStarted","Data":"f16436add7df7959112728734f2866f2de98919723f12e1136115f7baa2e8fd7"} Oct 31 21:29:20.047137 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:20.046949 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" event={"ID":"7866c78e-aaae-4fdb-948e-9facecee1434","Type":"ContainerStarted","Data":"468c37880594ac27611a80312d35ff566a4cddb5bfe8fddce572f80a283b4f59"} Oct 31 21:29:20.048488 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:20.048458 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" event={"ID":"8bde2fde-9142-40c6-bebc-22e38137ae60","Type":"ContainerStarted","Data":"8f4805577180f6203ff1ebc92c8dac2d2ca79f35615d5b6f10c0fe28abe80bbe"} Oct 31 21:29:20.893058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:20.893009 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 21:29:20.893255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:20.893081 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 21:29:21.058471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:21.058427 5188 generic.go:356] "Generic (PLEG): container finished" podID="7866c78e-aaae-4fdb-948e-9facecee1434" containerID="f16436add7df7959112728734f2866f2de98919723f12e1136115f7baa2e8fd7" exitCode=0 Oct 31 21:29:21.058915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:21.058504 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" event={"ID":"7866c78e-aaae-4fdb-948e-9facecee1434","Type":"ContainerDied","Data":"f16436add7df7959112728734f2866f2de98919723f12e1136115f7baa2e8fd7"} Oct 31 21:29:21.060466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:21.060436 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" event={"ID":"8bde2fde-9142-40c6-bebc-22e38137ae60","Type":"ContainerStarted","Data":"308e22d57f4a1c5a6c904614ad3eaddb0d6d3c58307c9e45519255759d554c59"} Oct 31 21:29:21.137886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:21.137830 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" podStartSLOduration=2.137814861 podStartE2EDuration="2.137814861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:21.135929038 +0000 UTC m=+807.828886432" watchObservedRunningTime="2026-10-31 21:29:21.137814861 +0000 UTC m=+807.830772250" Oct 31 21:29:22.069583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.069544 5188 generic.go:356] "Generic (PLEG): container finished" podID="8bde2fde-9142-40c6-bebc-22e38137ae60" containerID="308e22d57f4a1c5a6c904614ad3eaddb0d6d3c58307c9e45519255759d554c59" exitCode=0 Oct 31 21:29:22.070075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.069624 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" event={"ID":"8bde2fde-9142-40c6-bebc-22e38137ae60","Type":"ContainerDied","Data":"308e22d57f4a1c5a6c904614ad3eaddb0d6d3c58307c9e45519255759d554c59"} Oct 31 21:29:22.307121 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.307087 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:22.385672 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.385639 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:22.387089 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.387065 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:22.389365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.389050 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:29:22.391756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.391726 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7866c78e-aaae-4fdb-948e-9facecee1434-kube-api-access-bfpt2\" (UniqueName: \"kubernetes.io/projected/7866c78e-aaae-4fdb-948e-9facecee1434-kube-api-access-bfpt2\") pod \"7866c78e-aaae-4fdb-948e-9facecee1434\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " Oct 31 21:29:22.396777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.396748 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-openshift-ca\") pod \"7866c78e-aaae-4fdb-948e-9facecee1434\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " Oct 31 21:29:22.399126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.397256 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-openshift-ca" pod "7866c78e-aaae-4fdb-948e-9facecee1434" (UID: "7866c78e-aaae-4fdb-948e-9facecee1434"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:29:22.399570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.399549 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-xks-ca\") pod \"7866c78e-aaae-4fdb-948e-9facecee1434\" (UID: \"7866c78e-aaae-4fdb-948e-9facecee1434\") " Oct 31 21:29:22.400822 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.400798 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:22.403662 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.403432 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-xks-ca" pod "7866c78e-aaae-4fdb-948e-9facecee1434" (UID: "7866c78e-aaae-4fdb-948e-9facecee1434"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:29:22.408891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.408860 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7866c78e-aaae-4fdb-948e-9facecee1434-kube-api-access-bfpt2" pod "7866c78e-aaae-4fdb-948e-9facecee1434" (UID: "7866c78e-aaae-4fdb-948e-9facecee1434"). InnerVolumeSpecName "kube-api-access-bfpt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:29:22.502554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.502519 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfpt2\" (UniqueName: \"kubernetes.io/projected/7866c78e-aaae-4fdb-948e-9facecee1434-kube-api-access-bfpt2\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:22.502554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:22.502554 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7866c78e-aaae-4fdb-948e-9facecee1434-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:23.079457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.079424 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" Oct 31 21:29:23.079457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.079441 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl" event={"ID":"7866c78e-aaae-4fdb-948e-9facecee1434","Type":"ContainerDied","Data":"468c37880594ac27611a80312d35ff566a4cddb5bfe8fddce572f80a283b4f59"} Oct 31 21:29:23.080016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.079517 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="468c37880594ac27611a80312d35ff566a4cddb5bfe8fddce572f80a283b4f59" Oct 31 21:29:23.316366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.315262 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:23.425352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.424151 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-xks-ca\") pod \"8bde2fde-9142-40c6-bebc-22e38137ae60\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " Oct 31 21:29:23.425352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.424211 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bde2fde-9142-40c6-bebc-22e38137ae60-kube-api-access-dpxbn\" (UniqueName: \"kubernetes.io/projected/8bde2fde-9142-40c6-bebc-22e38137ae60-kube-api-access-dpxbn\") pod \"8bde2fde-9142-40c6-bebc-22e38137ae60\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " Oct 31 21:29:23.434283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.427473 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-openshift-ca\") pod \"8bde2fde-9142-40c6-bebc-22e38137ae60\" (UID: \"8bde2fde-9142-40c6-bebc-22e38137ae60\") " Oct 31 21:29:23.434283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.427867 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-openshift-ca" pod "8bde2fde-9142-40c6-bebc-22e38137ae60" (UID: "8bde2fde-9142-40c6-bebc-22e38137ae60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:29:23.434283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.429544 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:23.457582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.457534 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-xks-ca" pod "8bde2fde-9142-40c6-bebc-22e38137ae60" (UID: "8bde2fde-9142-40c6-bebc-22e38137ae60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:29:23.459875 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.459838 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bde2fde-9142-40c6-bebc-22e38137ae60-kube-api-access-dpxbn" pod "8bde2fde-9142-40c6-bebc-22e38137ae60" (UID: "8bde2fde-9142-40c6-bebc-22e38137ae60"). InnerVolumeSpecName "kube-api-access-dpxbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:29:23.531376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.531270 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bde2fde-9142-40c6-bebc-22e38137ae60-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:23.531376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:23.531317 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpxbn\" (UniqueName: \"kubernetes.io/projected/8bde2fde-9142-40c6-bebc-22e38137ae60-kube-api-access-dpxbn\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:24.090849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:24.090810 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" Oct 31 21:29:24.090849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:24.090817 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb" event={"ID":"8bde2fde-9142-40c6-bebc-22e38137ae60","Type":"ContainerDied","Data":"8f4805577180f6203ff1ebc92c8dac2d2ca79f35615d5b6f10c0fe28abe80bbe"} Oct 31 21:29:24.091473 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:24.090888 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f4805577180f6203ff1ebc92c8dac2d2ca79f35615d5b6f10c0fe28abe80bbe" Oct 31 21:29:29.039710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:29.039683 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:32.534520 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:32.534475 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:29:32.535162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:32.535110 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:29:32.652923 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:29:32.652879 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:29:35.072258 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.072221 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6"] Oct 31 21:29:35.072745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.072730 5188 cpu_manager.go:414] "containerMap: removing container" podUID="8bde2fde-9142-40c6-bebc-22e38137ae60" containerName="revoke-keys" Oct 31 21:29:35.072785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.072746 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bde2fde-9142-40c6-bebc-22e38137ae60" containerName="revoke-keys" Oct 31 21:29:35.072785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.072775 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7866c78e-aaae-4fdb-948e-9facecee1434" containerName="revoke-keys" Oct 31 21:29:35.072785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.072780 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7866c78e-aaae-4fdb-948e-9facecee1434" containerName="revoke-keys" Oct 31 21:29:35.072882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.072875 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="7866c78e-aaae-4fdb-948e-9facecee1434" containerName="revoke-keys" Oct 31 21:29:35.072915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.072888 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bde2fde-9142-40c6-bebc-22e38137ae60" containerName="revoke-keys" Oct 31 21:29:35.077550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.077531 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.096555 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.096526 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6"] Oct 31 21:29:35.144691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.144642 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-plugins-config-volume\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.144877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.144700 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t27l\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-kube-api-access-8t27l\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.144877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.144830 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-provider-credentials\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.144877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.144873 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-routing-overlay-volume\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.246377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.246336 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-plugins-config-volume\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.246609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.246398 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t27l\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-kube-api-access-8t27l\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.246609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.246479 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-provider-credentials\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.246609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.246517 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-routing-overlay-volume\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.247213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.247189 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-routing-overlay-volume\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.247349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.247259 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-plugins-config-volume\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.249389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.249361 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-provider-credentials\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.271137 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.268748 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t27l\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-kube-api-access-8t27l\") pod \"payload-processing-external-model-6b878ffdf9-qk8f6\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.391733 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.391691 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:35.653452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.653032 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk"] Oct 31 21:29:35.660710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.660677 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.687176 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.687134 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk"] Oct 31 21:29:35.719894 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.719853 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6"] Oct 31 21:29:35.750540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.750514 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgln5\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-kube-api-access-jgln5\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.750661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.750563 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.750661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.750588 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.750732 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.750680 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.750732 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.750700 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.750795 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.750774 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.750827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.750817 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.750905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.750892 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.750940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.750919 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.851835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.851801 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.852015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.851891 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.852288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.851927 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.852422 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.852314 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.852422 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.852402 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgln5\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-kube-api-access-jgln5\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.852536 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.852462 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.852793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.852770 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.852903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.852770 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.852903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.852499 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.852903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.852887 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.853058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.852919 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.853058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.852946 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.853280 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.853259 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.853547 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.853524 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.855634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.855601 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.855812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.855795 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.875389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.875354 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.875598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.875575 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgln5\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-kube-api-access-jgln5\") pod \"maas-default-gateway-openshift-default-58847f9fc5-f9pgk\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:35.981249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:35.981154 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:36.232572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:36.232480 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" event={"ID":"0ce2b6f8-5808-4ee5-841b-95a935675877","Type":"ContainerStarted","Data":"2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259"} Oct 31 21:29:36.232953 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:36.232711 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:36.232953 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:36.232748 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" event={"ID":"0ce2b6f8-5808-4ee5-841b-95a935675877","Type":"ContainerStarted","Data":"e4083c0d4b35f5ed1a41101f0203204b687d70f12a3b3548161e1e323b4628a0"} Oct 31 21:29:36.272560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:36.272499 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" podStartSLOduration=1.27248432 podStartE2EDuration="1.27248432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:36.269862151 +0000 UTC m=+822.962819543" watchObservedRunningTime="2026-10-31 21:29:36.27248432 +0000 UTC m=+822.965441711" Oct 31 21:29:36.305383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:36.305335 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk"] Oct 31 21:29:36.307980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:36.307939 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:29:36.308091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:36.308023 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:29:36.308091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:36.308059 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:29:37.241299 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:37.241261 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" event={"ID":"4021eab3-c6e4-47d6-93b3-1c2178324bc0","Type":"ContainerStarted","Data":"3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d"} Oct 31 21:29:37.241814 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:37.241314 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" event={"ID":"4021eab3-c6e4-47d6-93b3-1c2178324bc0","Type":"ContainerStarted","Data":"7bda7e2d476ab26f91ff943f371cbba8463711cf9e7edc5e403129182118321b"} Oct 31 21:29:37.992347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:37.989686 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:37.999481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:37.999442 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f9pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 21:29:37.999665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:37.999505 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 21:29:38.983358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:38.982790 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f9pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 21:29:38.983358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:38.982876 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 21:29:39.985349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:39.984785 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f9pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 21:29:39.985349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:39.984851 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 21:29:40.993351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:40.984844 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f9pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 21:29:40.993351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:40.984924 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 21:29:41.986049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:41.986004 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f9pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 21:29:41.986310 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:41.986279 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 21:29:42.983794 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:42.983754 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f9pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 21:29:42.984516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:42.984477 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 21:29:44.006355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:44.002346 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f9pgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 21:29:44.006355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:44.002413 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 21:29:45.720672 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:45.720637 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:45.721154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:45.720945 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:45.722309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:45.722283 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:29:45.810585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:45.810523 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podStartSLOduration=10.810508915 podStartE2EDuration="10.810508915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:37.300883377 +0000 UTC m=+823.993840769" watchObservedRunningTime="2026-10-31 21:29:45.810508915 +0000 UTC m=+832.503466305" Oct 31 21:29:47.244845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:47.244816 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:29:47.331893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:47.331857 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg"] Oct 31 21:29:52.332635 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.332593 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" podUID="dba98a77-478c-494f-8026-3a67d2ad039c" containerName="payload-processing" containerID="cri-o://cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16" gracePeriod=25 Oct 31 21:29:52.426197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.426161 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc"] Oct 31 21:29:52.430613 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.430585 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.458183 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.458155 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc"] Oct 31 21:29:52.500118 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.500077 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-provider-credentials\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.501177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.501147 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-plugins-config-volume\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.501369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.501201 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-routing-overlay-volume\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.501369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.501227 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kht4h\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-kube-api-access-kht4h\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.603108 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.603004 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-plugins-config-volume\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.603108 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.603073 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-routing-overlay-volume\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.603384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.603198 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kht4h\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-kube-api-access-kht4h\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.603384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.603359 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-provider-credentials\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.604068 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.604041 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-plugins-config-volume\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.604650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.604624 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-routing-overlay-volume\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.606692 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.606665 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-provider-credentials\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.625052 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.625012 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kht4h\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-kube-api-access-kht4h\") pod \"payload-processing-external-model-9888fbfd6-ggrmc\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.648415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.648382 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:52.704412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.704378 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-plugins-config-volume\") pod \"dba98a77-478c-494f-8026-3a67d2ad039c\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " Oct 31 21:29:52.704606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.704485 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-kube-api-access-l6vmb\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-kube-api-access-l6vmb\") pod \"dba98a77-478c-494f-8026-3a67d2ad039c\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " Oct 31 21:29:52.704606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.704526 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-routing-overlay-volume\") pod \"dba98a77-478c-494f-8026-3a67d2ad039c\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " Oct 31 21:29:52.704606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.704582 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-provider-credentials\") pod \"dba98a77-478c-494f-8026-3a67d2ad039c\" (UID: \"dba98a77-478c-494f-8026-3a67d2ad039c\") " Oct 31 21:29:52.705163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.705116 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-routing-overlay-volume" pod "dba98a77-478c-494f-8026-3a67d2ad039c" (UID: "dba98a77-478c-494f-8026-3a67d2ad039c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:29:52.705622 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.705543 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-plugins-config-volume" pod "dba98a77-478c-494f-8026-3a67d2ad039c" (UID: "dba98a77-478c-494f-8026-3a67d2ad039c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:29:52.707773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.707743 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-provider-credentials" pod "dba98a77-478c-494f-8026-3a67d2ad039c" (UID: "dba98a77-478c-494f-8026-3a67d2ad039c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:29:52.708120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.708098 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-kube-api-access-l6vmb" pod "dba98a77-478c-494f-8026-3a67d2ad039c" (UID: "dba98a77-478c-494f-8026-3a67d2ad039c"). InnerVolumeSpecName "kube-api-access-l6vmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:29:52.744797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.744765 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:52.806456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.806424 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6vmb\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-kube-api-access-l6vmb\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:52.806456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.806456 5188 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-routing-overlay-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:52.806616 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.806465 5188 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dba98a77-478c-494f-8026-3a67d2ad039c-provider-credentials\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:52.806616 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:52.806474 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dba98a77-478c-494f-8026-3a67d2ad039c-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:29:53.050357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.050293 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc"] Oct 31 21:29:53.050357 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:29:53.050336 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc784bd92_84dd_489a_9b6d_5ef6b5dcf5af.slice/crio-8ba194fe5b8c4bd817266784766d6b003374aec0691cae2880117672226a7ddc WatchSource:0}: Error finding container 8ba194fe5b8c4bd817266784766d6b003374aec0691cae2880117672226a7ddc: Status 404 returned error can't find the container with id 8ba194fe5b8c4bd817266784766d6b003374aec0691cae2880117672226a7ddc Oct 31 21:29:53.440782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.440740 5188 generic.go:356] "Generic (PLEG): container finished" podID="dba98a77-478c-494f-8026-3a67d2ad039c" containerID="cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16" exitCode=0 Oct 31 21:29:53.441199 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.440792 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" Oct 31 21:29:53.441199 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.440813 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" event={"ID":"dba98a77-478c-494f-8026-3a67d2ad039c","Type":"ContainerDied","Data":"cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16"} Oct 31 21:29:53.441199 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.440868 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg" event={"ID":"dba98a77-478c-494f-8026-3a67d2ad039c","Type":"ContainerDied","Data":"1c874d4fbed669e2cd79d72a0421ed3f3bfdb96829aadb2826a324ffd6e9b02f"} Oct 31 21:29:53.441199 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.440896 5188 scope.go:122] "RemoveContainer" containerID="cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16" Oct 31 21:29:53.442752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.442683 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" event={"ID":"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af","Type":"ContainerStarted","Data":"ad20a79f870def77dd0bf54a1068a328ff397b9357b6f8d029f34c37eef0d148"} Oct 31 21:29:53.442752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.442744 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" event={"ID":"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af","Type":"ContainerStarted","Data":"8ba194fe5b8c4bd817266784766d6b003374aec0691cae2880117672226a7ddc"} Oct 31 21:29:53.442970 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.442768 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:29:53.451719 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.451698 5188 scope.go:122] "RemoveContainer" containerID="cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16" Oct 31 21:29:53.452040 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:29:53.452020 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16\": container with ID starting with cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16 not found: ID does not exist" containerID="cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16" Oct 31 21:29:53.452102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.452049 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16"} err="failed to get container status \"cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16\": rpc error: code = NotFound desc = could not find container \"cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16\": container with ID starting with cbbc43bf49ffc195356f599171b291a923d1fce2dd5019ffa99e97aea9a41b16 not found: ID does not exist" Oct 31 21:29:53.487680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.487531 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg"] Oct 31 21:29:53.501081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.501045 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78fb45f9df-z72zg"] Oct 31 21:29:53.533716 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:53.533646 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" podStartSLOduration=1.5336249990000002 podStartE2EDuration="1.533624999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:53.532551277 +0000 UTC m=+840.225508670" watchObservedRunningTime="2026-10-31 21:29:53.533624999 +0000 UTC m=+840.226582391" Oct 31 21:29:55.472854 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:29:55.472819 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba98a77-478c-494f-8026-3a67d2ad039c" path="/var/lib/kubelet/pods/dba98a77-478c-494f-8026-3a67d2ad039c/volumes" Oct 31 21:30:00.189348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.188440 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s"] Oct 31 21:30:00.189348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.189306 5188 cpu_manager.go:414] "containerMap: removing container" podUID="dba98a77-478c-494f-8026-3a67d2ad039c" containerName="payload-processing" Oct 31 21:30:00.189348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.189343 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba98a77-478c-494f-8026-3a67d2ad039c" containerName="payload-processing" Oct 31 21:30:00.190011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.189493 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba98a77-478c-494f-8026-3a67d2ad039c" containerName="payload-processing" Oct 31 21:30:00.212031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.207024 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j"] Oct 31 21:30:00.212031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.209936 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:00.282448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.224576 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:30:00.282448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.224698 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:30:00.282448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.243012 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:00.282448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.248404 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f"] Oct 31 21:30:00.282448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.259710 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:00.282448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.267556 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v"] Oct 31 21:30:00.282448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.272278 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s"] Oct 31 21:30:00.282448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.272387 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:00.348871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.296869 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:30:00.348871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.346347 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9593b80e-95b7-4454-ba3f-f7828865fbc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s\" (UID: \"9593b80e-95b7-4454-ba3f-f7828865fbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:00.348871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.346398 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913eb467-04c5-41f0-a26f-8e8739eb0277-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891370-fb42f\" (UID: \"913eb467-04c5-41f0-a26f-8e8739eb0277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:00.348871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.346439 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqvv9\" (UniqueName: \"kubernetes.io/projected/913eb467-04c5-41f0-a26f-8e8739eb0277-kube-api-access-kqvv9\") pod \"maas-api-key-cleanup-29891370-fb42f\" (UID: \"913eb467-04c5-41f0-a26f-8e8739eb0277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:00.348871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.346495 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a8cf4f-0e49-40c9-bcae-9272cb45a364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j\" (UID: \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:00.348871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.346587 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dch8\" (UniqueName: \"kubernetes.io/projected/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-kube-api-access-2dch8\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v\" (UID: \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:00.348871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.346638 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl4fg\" (UniqueName: \"kubernetes.io/projected/24a8cf4f-0e49-40c9-bcae-9272cb45a364-kube-api-access-rl4fg\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j\" (UID: \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:00.348871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.346720 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v\" (UID: \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:00.348871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.346798 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fr4h\" (UniqueName: \"kubernetes.io/projected/9593b80e-95b7-4454-ba3f-f7828865fbc9-kube-api-access-9fr4h\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s\" (UID: \"9593b80e-95b7-4454-ba3f-f7828865fbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.452987 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4fg\" (UniqueName: \"kubernetes.io/projected/24a8cf4f-0e49-40c9-bcae-9272cb45a364-kube-api-access-rl4fg\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j\" (UID: \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.453101 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v\" (UID: \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.453179 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fr4h\" (UniqueName: \"kubernetes.io/projected/9593b80e-95b7-4454-ba3f-f7828865fbc9-kube-api-access-9fr4h\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s\" (UID: \"9593b80e-95b7-4454-ba3f-f7828865fbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.453240 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9593b80e-95b7-4454-ba3f-f7828865fbc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s\" (UID: \"9593b80e-95b7-4454-ba3f-f7828865fbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.453269 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913eb467-04c5-41f0-a26f-8e8739eb0277-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891370-fb42f\" (UID: \"913eb467-04c5-41f0-a26f-8e8739eb0277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.453302 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqvv9\" (UniqueName: \"kubernetes.io/projected/913eb467-04c5-41f0-a26f-8e8739eb0277-kube-api-access-kqvv9\") pod \"maas-api-key-cleanup-29891370-fb42f\" (UID: \"913eb467-04c5-41f0-a26f-8e8739eb0277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.453382 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a8cf4f-0e49-40c9-bcae-9272cb45a364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j\" (UID: \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.453472 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dch8\" (UniqueName: \"kubernetes.io/projected/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-kube-api-access-2dch8\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v\" (UID: \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.454963 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v\" (UID: \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.455963 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9593b80e-95b7-4454-ba3f-f7828865fbc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s\" (UID: \"9593b80e-95b7-4454-ba3f-f7828865fbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:00.526406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.476812 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a8cf4f-0e49-40c9-bcae-9272cb45a364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j\" (UID: \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:00.575408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.530896 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913eb467-04c5-41f0-a26f-8e8739eb0277-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891370-fb42f\" (UID: \"913eb467-04c5-41f0-a26f-8e8739eb0277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:00.575408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.556781 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j"] Oct 31 21:30:00.575408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.558836 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f"] Oct 31 21:30:00.575408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.564733 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v"] Oct 31 21:30:00.922561 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.917627 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqvv9\" (UniqueName: \"kubernetes.io/projected/913eb467-04c5-41f0-a26f-8e8739eb0277-kube-api-access-kqvv9\") pod \"maas-api-key-cleanup-29891370-fb42f\" (UID: \"913eb467-04c5-41f0-a26f-8e8739eb0277\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:00.943296 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.941630 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dch8\" (UniqueName: \"kubernetes.io/projected/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-kube-api-access-2dch8\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v\" (UID: \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:01.050646 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:00.919467 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4fg\" (UniqueName: \"kubernetes.io/projected/24a8cf4f-0e49-40c9-bcae-9272cb45a364-kube-api-access-rl4fg\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j\" (UID: \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:01.320376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:01.210126 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fr4h\" (UniqueName: \"kubernetes.io/projected/9593b80e-95b7-4454-ba3f-f7828865fbc9-kube-api-access-9fr4h\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s\" (UID: \"9593b80e-95b7-4454-ba3f-f7828865fbc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:01.320376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:01.294143 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:01.413349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:01.412887 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:01.486714 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:01.486679 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:01.601266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:01.492938 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:01.601266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:01.497115 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:01.601266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:01.500714 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:01.911223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:01.910809 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f9pgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:30:01.911223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:01.910895 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:02.475917 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:02.475884 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9593b80e_95b7_4454_ba3f_f7828865fbc9.slice/crio-74fb6b4724907b9ef24568e2bbed162d3635b24e368abc0f584b948a6b8ee056 WatchSource:0}: Error finding container 74fb6b4724907b9ef24568e2bbed162d3635b24e368abc0f584b948a6b8ee056: Status 404 returned error can't find the container with id 74fb6b4724907b9ef24568e2bbed162d3635b24e368abc0f584b948a6b8ee056 Oct 31 21:30:02.478256 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:02.478227 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s"] Oct 31 21:30:02.540978 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:02.540940 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:30:02.553659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:02.553627 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v"] Oct 31 21:30:02.554449 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:02.554420 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb718d7b4_9366_46b2_88a4_41dddfe3c7a6.slice/crio-15953d5f953c295ac93af77383eb46603222fb1aab857557b7bb1ebdf1b27c65 WatchSource:0}: Error finding container 15953d5f953c295ac93af77383eb46603222fb1aab857557b7bb1ebdf1b27c65: Status 404 returned error can't find the container with id 15953d5f953c295ac93af77383eb46603222fb1aab857557b7bb1ebdf1b27c65 Oct 31 21:30:02.610419 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:02.610391 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f"] Oct 31 21:30:02.640104 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:02.640081 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j"] Oct 31 21:30:02.674900 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:02.674866 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913eb467_04c5_41f0_a26f_8e8739eb0277.slice/crio-da41b7a40f4095d16fa3364439175cf00cec9d1385718da51aad775a94b794b7 WatchSource:0}: Error finding container da41b7a40f4095d16fa3364439175cf00cec9d1385718da51aad775a94b794b7: Status 404 returned error can't find the container with id da41b7a40f4095d16fa3364439175cf00cec9d1385718da51aad775a94b794b7 Oct 31 21:30:02.676164 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:02.676137 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a8cf4f_0e49_40c9_bcae_9272cb45a364.slice/crio-f9011c74d12e7f7fc91247095b9b7e785737dfc71f11220616391672f0752b26 WatchSource:0}: Error finding container f9011c74d12e7f7fc91247095b9b7e785737dfc71f11220616391672f0752b26: Status 404 returned error can't find the container with id f9011c74d12e7f7fc91247095b9b7e785737dfc71f11220616391672f0752b26 Oct 31 21:30:03.444981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.444939 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" event={"ID":"24a8cf4f-0e49-40c9-bcae-9272cb45a364","Type":"ContainerStarted","Data":"b334f7dcb2334551ea9925bb7119043824205dab795a9d10bd55f6088df1eed7"} Oct 31 21:30:03.445185 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.445003 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" event={"ID":"24a8cf4f-0e49-40c9-bcae-9272cb45a364","Type":"ContainerStarted","Data":"f9011c74d12e7f7fc91247095b9b7e785737dfc71f11220616391672f0752b26"} Oct 31 21:30:03.446566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.446536 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" event={"ID":"913eb467-04c5-41f0-a26f-8e8739eb0277","Type":"ContainerStarted","Data":"71bceab10d2478ef18122e24e1279715eef7c0619a598e9d858f3e54f9316008"} Oct 31 21:30:03.446688 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.446590 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" event={"ID":"913eb467-04c5-41f0-a26f-8e8739eb0277","Type":"ContainerStarted","Data":"da41b7a40f4095d16fa3364439175cf00cec9d1385718da51aad775a94b794b7"} Oct 31 21:30:03.448187 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.448154 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" event={"ID":"9593b80e-95b7-4454-ba3f-f7828865fbc9","Type":"ContainerStarted","Data":"3d050938382a6aef8b4da216a7354d1980594c62a411baab7dee0ae4f28bba65"} Oct 31 21:30:03.448300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.448204 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" event={"ID":"9593b80e-95b7-4454-ba3f-f7828865fbc9","Type":"ContainerStarted","Data":"74fb6b4724907b9ef24568e2bbed162d3635b24e368abc0f584b948a6b8ee056"} Oct 31 21:30:03.449656 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.449630 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" event={"ID":"b718d7b4-9366-46b2-88a4-41dddfe3c7a6","Type":"ContainerStarted","Data":"eaf0c4cb0b95a7ffd52bb31681a99a5047c80ddafc542ee01f676f1f121c79ca"} Oct 31 21:30:03.449752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.449671 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" event={"ID":"b718d7b4-9366-46b2-88a4-41dddfe3c7a6","Type":"ContainerStarted","Data":"15953d5f953c295ac93af77383eb46603222fb1aab857557b7bb1ebdf1b27c65"} Oct 31 21:30:03.483141 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.483085 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" podStartSLOduration=3.483071581 podStartE2EDuration="3.483071581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:03.481810093 +0000 UTC m=+850.174767484" watchObservedRunningTime="2026-10-31 21:30:03.483071581 +0000 UTC m=+850.176028998" Oct 31 21:30:03.524487 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.524433 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" podStartSLOduration=3.5244181660000002 podStartE2EDuration="3.524418166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:03.521689876 +0000 UTC m=+850.214647268" watchObservedRunningTime="2026-10-31 21:30:03.524418166 +0000 UTC m=+850.217375560" Oct 31 21:30:03.560709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.560653 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" podStartSLOduration=3.560638564 podStartE2EDuration="3.560638564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:03.556880476 +0000 UTC m=+850.249837868" watchObservedRunningTime="2026-10-31 21:30:03.560638564 +0000 UTC m=+850.253595956" Oct 31 21:30:03.596807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:03.596745 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" podStartSLOduration=3.596724227 podStartE2EDuration="3.596724227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:03.592629467 +0000 UTC m=+850.285586863" watchObservedRunningTime="2026-10-31 21:30:03.596724227 +0000 UTC m=+850.289681621" Oct 31 21:30:04.454151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.454120 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:30:04.458166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.458131 5188 generic.go:356] "Generic (PLEG): container finished" podID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerID="eaf0c4cb0b95a7ffd52bb31681a99a5047c80ddafc542ee01f676f1f121c79ca" exitCode=60 Oct 31 21:30:04.458368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.458215 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" event={"ID":"b718d7b4-9366-46b2-88a4-41dddfe3c7a6","Type":"ContainerDied","Data":"eaf0c4cb0b95a7ffd52bb31681a99a5047c80ddafc542ee01f676f1f121c79ca"} Oct 31 21:30:04.458510 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.458493 5188 scope.go:122] "RemoveContainer" containerID="eaf0c4cb0b95a7ffd52bb31681a99a5047c80ddafc542ee01f676f1f121c79ca" Oct 31 21:30:04.460214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.460195 5188 generic.go:356] "Generic (PLEG): container finished" podID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerID="b334f7dcb2334551ea9925bb7119043824205dab795a9d10bd55f6088df1eed7" exitCode=60 Oct 31 21:30:04.460308 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.460284 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" event={"ID":"24a8cf4f-0e49-40c9-bcae-9272cb45a364","Type":"ContainerDied","Data":"b334f7dcb2334551ea9925bb7119043824205dab795a9d10bd55f6088df1eed7"} Oct 31 21:30:04.460532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.460518 5188 scope.go:122] "RemoveContainer" containerID="b334f7dcb2334551ea9925bb7119043824205dab795a9d10bd55f6088df1eed7" Oct 31 21:30:04.462211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.462190 5188 generic.go:356] "Generic (PLEG): container finished" podID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerID="71bceab10d2478ef18122e24e1279715eef7c0619a598e9d858f3e54f9316008" exitCode=60 Oct 31 21:30:04.462337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.462270 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" event={"ID":"913eb467-04c5-41f0-a26f-8e8739eb0277","Type":"ContainerDied","Data":"71bceab10d2478ef18122e24e1279715eef7c0619a598e9d858f3e54f9316008"} Oct 31 21:30:04.462496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.462480 5188 scope.go:122] "RemoveContainer" containerID="71bceab10d2478ef18122e24e1279715eef7c0619a598e9d858f3e54f9316008" Oct 31 21:30:04.463952 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.463928 5188 generic.go:356] "Generic (PLEG): container finished" podID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerID="3d050938382a6aef8b4da216a7354d1980594c62a411baab7dee0ae4f28bba65" exitCode=60 Oct 31 21:30:04.464038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.463991 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" event={"ID":"9593b80e-95b7-4454-ba3f-f7828865fbc9","Type":"ContainerDied","Data":"3d050938382a6aef8b4da216a7354d1980594c62a411baab7dee0ae4f28bba65"} Oct 31 21:30:04.464207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.464194 5188 scope.go:122] "RemoveContainer" containerID="3d050938382a6aef8b4da216a7354d1980594c62a411baab7dee0ae4f28bba65" Oct 31 21:30:04.546799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:04.546768 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6"] Oct 31 21:30:05.477869 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:05.477273 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" event={"ID":"913eb467-04c5-41f0-a26f-8e8739eb0277","Type":"ContainerStarted","Data":"e917dd2f944a8330bb22b906798ce5c78e5af490183c599ea88c475452e7d4b1"} Oct 31 21:30:05.480301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:05.480267 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" event={"ID":"9593b80e-95b7-4454-ba3f-f7828865fbc9","Type":"ContainerStarted","Data":"ad9f57d85ef70bd4bf2d2435814fb8582bbee206ee55c68e36d89bfc7dd564df"} Oct 31 21:30:05.483514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:05.483483 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" event={"ID":"b718d7b4-9366-46b2-88a4-41dddfe3c7a6","Type":"ContainerStarted","Data":"7ada70aa0ff5016bef1aa73eb0d431af02528f7fb3bad04f15162ada29ee498f"} Oct 31 21:30:05.485642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:05.485619 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" event={"ID":"24a8cf4f-0e49-40c9-bcae-9272cb45a364","Type":"ContainerStarted","Data":"cb568de638b51aec4aabba795d02111c841527e6f984e58b1db76fbae26bbd7e"} Oct 31 21:30:06.510547 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.510488 5188 generic.go:356] "Generic (PLEG): container finished" podID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerID="cb568de638b51aec4aabba795d02111c841527e6f984e58b1db76fbae26bbd7e" exitCode=60 Oct 31 21:30:06.511028 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.510646 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" event={"ID":"24a8cf4f-0e49-40c9-bcae-9272cb45a364","Type":"ContainerDied","Data":"cb568de638b51aec4aabba795d02111c841527e6f984e58b1db76fbae26bbd7e"} Oct 31 21:30:06.511028 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.510751 5188 scope.go:122] "RemoveContainer" containerID="b334f7dcb2334551ea9925bb7119043824205dab795a9d10bd55f6088df1eed7" Oct 31 21:30:06.512745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.512430 5188 scope.go:122] "RemoveContainer" containerID="cb568de638b51aec4aabba795d02111c841527e6f984e58b1db76fbae26bbd7e" Oct 31 21:30:06.513141 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:06.513090 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j_odh-ai-gateway-infra(24a8cf4f-0e49-40c9-bcae-9272cb45a364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" Oct 31 21:30:06.516980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.516948 5188 generic.go:356] "Generic (PLEG): container finished" podID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerID="e917dd2f944a8330bb22b906798ce5c78e5af490183c599ea88c475452e7d4b1" exitCode=60 Oct 31 21:30:06.517100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.517022 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" event={"ID":"913eb467-04c5-41f0-a26f-8e8739eb0277","Type":"ContainerDied","Data":"e917dd2f944a8330bb22b906798ce5c78e5af490183c599ea88c475452e7d4b1"} Oct 31 21:30:06.517555 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.517400 5188 scope.go:122] "RemoveContainer" containerID="e917dd2f944a8330bb22b906798ce5c78e5af490183c599ea88c475452e7d4b1" Oct 31 21:30:06.517710 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:06.517676 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891370-fb42f_odh-ai-gateway-infra(913eb467-04c5-41f0-a26f-8e8739eb0277)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" Oct 31 21:30:06.519774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.519752 5188 generic.go:356] "Generic (PLEG): container finished" podID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerID="ad9f57d85ef70bd4bf2d2435814fb8582bbee206ee55c68e36d89bfc7dd564df" exitCode=60 Oct 31 21:30:06.521783 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.520086 5188 scope.go:122] "RemoveContainer" containerID="ad9f57d85ef70bd4bf2d2435814fb8582bbee206ee55c68e36d89bfc7dd564df" Oct 31 21:30:06.521783 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:06.520367 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s_odh-ai-gateway-infra(9593b80e-95b7-4454-ba3f-f7828865fbc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" Oct 31 21:30:06.521783 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.521699 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" event={"ID":"9593b80e-95b7-4454-ba3f-f7828865fbc9","Type":"ContainerDied","Data":"ad9f57d85ef70bd4bf2d2435814fb8582bbee206ee55c68e36d89bfc7dd564df"} Oct 31 21:30:06.523803 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.523782 5188 generic.go:356] "Generic (PLEG): container finished" podID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerID="7ada70aa0ff5016bef1aa73eb0d431af02528f7fb3bad04f15162ada29ee498f" exitCode=60 Oct 31 21:30:06.523905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.523851 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" event={"ID":"b718d7b4-9366-46b2-88a4-41dddfe3c7a6","Type":"ContainerDied","Data":"7ada70aa0ff5016bef1aa73eb0d431af02528f7fb3bad04f15162ada29ee498f"} Oct 31 21:30:06.524155 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.524138 5188 scope.go:122] "RemoveContainer" containerID="7ada70aa0ff5016bef1aa73eb0d431af02528f7fb3bad04f15162ada29ee498f" Oct 31 21:30:06.524469 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:06.524446 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v_odh-ai-gateway-infra(b718d7b4-9366-46b2-88a4-41dddfe3c7a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" Oct 31 21:30:06.528052 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.528033 5188 scope.go:122] "RemoveContainer" containerID="71bceab10d2478ef18122e24e1279715eef7c0619a598e9d858f3e54f9316008" Oct 31 21:30:06.541595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.541572 5188 scope.go:122] "RemoveContainer" containerID="3d050938382a6aef8b4da216a7354d1980594c62a411baab7dee0ae4f28bba65" Oct 31 21:30:06.555376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:06.555358 5188 scope.go:122] "RemoveContainer" containerID="eaf0c4cb0b95a7ffd52bb31681a99a5047c80ddafc542ee01f676f1f121c79ca" Oct 31 21:30:07.534913 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:07.534872 5188 scope.go:122] "RemoveContainer" containerID="cb568de638b51aec4aabba795d02111c841527e6f984e58b1db76fbae26bbd7e" Oct 31 21:30:07.535465 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:07.535210 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j_odh-ai-gateway-infra(24a8cf4f-0e49-40c9-bcae-9272cb45a364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" Oct 31 21:30:07.536824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:07.536797 5188 scope.go:122] "RemoveContainer" containerID="e917dd2f944a8330bb22b906798ce5c78e5af490183c599ea88c475452e7d4b1" Oct 31 21:30:07.537133 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:07.537112 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891370-fb42f_odh-ai-gateway-infra(913eb467-04c5-41f0-a26f-8e8739eb0277)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" Oct 31 21:30:07.538639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:07.538616 5188 scope.go:122] "RemoveContainer" containerID="ad9f57d85ef70bd4bf2d2435814fb8582bbee206ee55c68e36d89bfc7dd564df" Oct 31 21:30:07.538904 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:07.538876 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s_odh-ai-gateway-infra(9593b80e-95b7-4454-ba3f-f7828865fbc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" Oct 31 21:30:07.540290 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:07.540270 5188 scope.go:122] "RemoveContainer" containerID="7ada70aa0ff5016bef1aa73eb0d431af02528f7fb3bad04f15162ada29ee498f" Oct 31 21:30:07.540544 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:07.540526 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v_odh-ai-gateway-infra(b718d7b4-9366-46b2-88a4-41dddfe3c7a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" Oct 31 21:30:09.547517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.547472 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" podUID="0ce2b6f8-5808-4ee5-841b-95a935675877" containerName="payload-processing" containerID="cri-o://2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259" gracePeriod=25 Oct 31 21:30:09.831203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.831175 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:30:09.891041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.891003 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-routing-overlay-volume\") pod \"0ce2b6f8-5808-4ee5-841b-95a935675877\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " Oct 31 21:30:09.891306 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.891057 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-kube-api-access-8t27l\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-kube-api-access-8t27l\") pod \"0ce2b6f8-5808-4ee5-841b-95a935675877\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " Oct 31 21:30:09.891306 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.891108 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-provider-credentials\") pod \"0ce2b6f8-5808-4ee5-841b-95a935675877\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " Oct 31 21:30:09.891306 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.891176 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-plugins-config-volume\") pod \"0ce2b6f8-5808-4ee5-841b-95a935675877\" (UID: \"0ce2b6f8-5808-4ee5-841b-95a935675877\") " Oct 31 21:30:09.891511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.891451 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-routing-overlay-volume" pod "0ce2b6f8-5808-4ee5-841b-95a935675877" (UID: "0ce2b6f8-5808-4ee5-841b-95a935675877"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:09.891880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.891854 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-plugins-config-volume" pod "0ce2b6f8-5808-4ee5-841b-95a935675877" (UID: "0ce2b6f8-5808-4ee5-841b-95a935675877"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:09.891980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.891916 5188 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-routing-overlay-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:09.894359 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.893969 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-provider-credentials" pod "0ce2b6f8-5808-4ee5-841b-95a935675877" (UID: "0ce2b6f8-5808-4ee5-841b-95a935675877"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:09.894359 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.894108 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-kube-api-access-8t27l" pod "0ce2b6f8-5808-4ee5-841b-95a935675877" (UID: "0ce2b6f8-5808-4ee5-841b-95a935675877"). InnerVolumeSpecName "kube-api-access-8t27l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:09.992694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.992660 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t27l\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-kube-api-access-8t27l\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:09.992694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.992692 5188 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ce2b6f8-5808-4ee5-841b-95a935675877-provider-credentials\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:09.992694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:09.992707 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce2b6f8-5808-4ee5-841b-95a935675877-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:10.564069 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:10.564033 5188 generic.go:356] "Generic (PLEG): container finished" podID="0ce2b6f8-5808-4ee5-841b-95a935675877" containerID="2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259" exitCode=0 Oct 31 21:30:10.564511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:10.564084 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" Oct 31 21:30:10.564511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:10.564117 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" event={"ID":"0ce2b6f8-5808-4ee5-841b-95a935675877","Type":"ContainerDied","Data":"2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259"} Oct 31 21:30:10.564511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:10.564172 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6" event={"ID":"0ce2b6f8-5808-4ee5-841b-95a935675877","Type":"ContainerDied","Data":"e4083c0d4b35f5ed1a41101f0203204b687d70f12a3b3548161e1e323b4628a0"} Oct 31 21:30:10.564511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:10.564194 5188 scope.go:122] "RemoveContainer" containerID="2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259" Oct 31 21:30:10.576889 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:10.576865 5188 scope.go:122] "RemoveContainer" containerID="2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259" Oct 31 21:30:10.577353 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:10.577297 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259\": container with ID starting with 2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259 not found: ID does not exist" containerID="2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259" Oct 31 21:30:10.577525 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:10.577363 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259"} err="failed to get container status \"2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259\": rpc error: code = NotFound desc = could not find container \"2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259\": container with ID starting with 2eaeb7aaf24f166ea75e818d9326d1e54aaa36dbf0c732125e3e90c309c6a259 not found: ID does not exist" Oct 31 21:30:10.613290 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:10.613257 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6"] Oct 31 21:30:10.622427 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:10.622397 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b878ffdf9-qk8f6"] Oct 31 21:30:11.475237 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:11.475192 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce2b6f8-5808-4ee5-841b-95a935675877" path="/var/lib/kubelet/pods/0ce2b6f8-5808-4ee5-841b-95a935675877/volumes" Oct 31 21:30:12.888899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.888859 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2"] Oct 31 21:30:12.890351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.890294 5188 cpu_manager.go:414] "containerMap: removing container" podUID="0ce2b6f8-5808-4ee5-841b-95a935675877" containerName="payload-processing" Oct 31 21:30:12.890351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.890343 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce2b6f8-5808-4ee5-841b-95a935675877" containerName="payload-processing" Oct 31 21:30:12.890551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.890535 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce2b6f8-5808-4ee5-841b-95a935675877" containerName="payload-processing" Oct 31 21:30:12.894880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.894856 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:12.907911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.907863 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-728ed92ccb8f\"" Oct 31 21:30:12.915924 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.915884 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2"] Oct 31 21:30:12.940817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.940777 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-plugins-config-volume\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:12.941049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.941023 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-routing-overlay-volume\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:12.941153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.941094 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-provider-credentials\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:12.941237 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:12.941223 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqcwz\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-kube-api-access-cqcwz\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.008773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.008741 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a8266d459040\"" Oct 31 21:30:13.042764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.042728 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-plugins-config-volume\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.042997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.042774 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-routing-overlay-volume\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.042997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.042831 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-provider-credentials\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.042997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.042898 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqcwz\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-kube-api-access-cqcwz\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.043606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.043586 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-routing-overlay-volume\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.043689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.043632 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-plugins-config-volume\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.045545 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.045515 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-provider-credentials\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.064274 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.064234 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqcwz\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-kube-api-access-cqcwz\") pod \"payload-processing-external-model-bb987f6c7-gdbp2\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.223208 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.223106 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:13.538182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.538142 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2"] Oct 31 21:30:13.539180 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:13.539152 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647513a8_89c8_465a_a7ca_825eefc87d63.slice/crio-52b643017a142b3b33aaa8035f996aa915058d8deaf5bcaa1a3ecbce54f49884 WatchSource:0}: Error finding container 52b643017a142b3b33aaa8035f996aa915058d8deaf5bcaa1a3ecbce54f49884: Status 404 returned error can't find the container with id 52b643017a142b3b33aaa8035f996aa915058d8deaf5bcaa1a3ecbce54f49884 Oct 31 21:30:13.613170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.613116 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" event={"ID":"647513a8-89c8-465a-a7ca-825eefc87d63","Type":"ContainerStarted","Data":"52b643017a142b3b33aaa8035f996aa915058d8deaf5bcaa1a3ecbce54f49884"} Oct 31 21:30:13.740651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.740610 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k"] Oct 31 21:30:13.744742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.744712 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.774360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.774293 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k"] Oct 31 21:30:13.853393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.853249 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.853393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.853374 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcxdb\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-kube-api-access-xcxdb\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.853615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.853419 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.853615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.853529 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.853694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.853629 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.853731 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.853701 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.853775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.853760 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.853880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.853861 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.853936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.853891 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.954991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.954951 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955002 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955043 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955079 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955099 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955127 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955177 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcxdb\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-kube-api-access-xcxdb\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955199 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955234 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955515 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955605 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955788 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955790 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.955910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.955872 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.957592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.957571 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.957848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.957831 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.976843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.976811 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:13.977029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:13.976920 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcxdb\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-kube-api-access-xcxdb\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:14.063175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.063130 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:14.398867 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:14.398828 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27c03e2_52d5_48b7_89c9_36bc18aa09db.slice/crio-42ed3aceee3e867aefd8d5dd7db78eafa5bdd2805a36b938979beb1fa5fdaaa4 WatchSource:0}: Error finding container 42ed3aceee3e867aefd8d5dd7db78eafa5bdd2805a36b938979beb1fa5fdaaa4: Status 404 returned error can't find the container with id 42ed3aceee3e867aefd8d5dd7db78eafa5bdd2805a36b938979beb1fa5fdaaa4 Oct 31 21:30:14.399002 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.398977 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k"] Oct 31 21:30:14.401426 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.401395 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:14.401509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.401464 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:14.401509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.401492 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:14.622800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.622760 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" event={"ID":"f27c03e2-52d5-48b7-89c9-36bc18aa09db","Type":"ContainerStarted","Data":"ecd68dde986329c1b4fa0abb255df0e0d7c69b92e7ce896e232388b6d38f998e"} Oct 31 21:30:14.623014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.622829 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" event={"ID":"f27c03e2-52d5-48b7-89c9-36bc18aa09db","Type":"ContainerStarted","Data":"42ed3aceee3e867aefd8d5dd7db78eafa5bdd2805a36b938979beb1fa5fdaaa4"} Oct 31 21:30:14.624797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.624766 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" event={"ID":"647513a8-89c8-465a-a7ca-825eefc87d63","Type":"ContainerStarted","Data":"7f0516881241fdcac84be5e1a1a554ce943d2fc0c950ddd6995beb34006fce26"} Oct 31 21:30:14.624949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.624848 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:14.668918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.668765 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podStartSLOduration=1.668746009 podStartE2EDuration="1.668746009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:14.665228003 +0000 UTC m=+861.358185395" watchObservedRunningTime="2026-10-31 21:30:14.668746009 +0000 UTC m=+861.361703405" Oct 31 21:30:14.713931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:14.713799 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" podStartSLOduration=2.713782286 podStartE2EDuration="2.713782286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:14.712872097 +0000 UTC m=+861.405829489" watchObservedRunningTime="2026-10-31 21:30:14.713782286 +0000 UTC m=+861.406739681" Oct 31 21:30:15.063920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:15.063832 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:15.065512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:15.065479 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:30:15.065647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:15.065531 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 21:30:16.064078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:16.064040 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:30:16.064536 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:16.064100 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 21:30:17.064177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:17.064137 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:30:17.064652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:17.064204 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 21:30:17.466300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:17.466256 5188 scope.go:122] "RemoveContainer" containerID="cb568de638b51aec4aabba795d02111c841527e6f984e58b1db76fbae26bbd7e" Oct 31 21:30:18.064655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:18.064558 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:30:18.064655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:18.064629 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 21:30:18.466114 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:18.466075 5188 scope.go:122] "RemoveContainer" containerID="7ada70aa0ff5016bef1aa73eb0d431af02528f7fb3bad04f15162ada29ee498f" Oct 31 21:30:18.466294 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:18.466208 5188 scope.go:122] "RemoveContainer" containerID="e917dd2f944a8330bb22b906798ce5c78e5af490183c599ea88c475452e7d4b1" Oct 31 21:30:18.657110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:18.657066 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" event={"ID":"24a8cf4f-0e49-40c9-bcae-9272cb45a364","Type":"ContainerStarted","Data":"8ba669bd84c81a3c1d3c0b8726badb26253b432a54c386034e16b8aa5de742fd"} Oct 31 21:30:19.064226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.064136 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:30:19.064226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.064194 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 21:30:19.465962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.465929 5188 scope.go:122] "RemoveContainer" containerID="ad9f57d85ef70bd4bf2d2435814fb8582bbee206ee55c68e36d89bfc7dd564df" Oct 31 21:30:19.668736 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.668701 5188 generic.go:356] "Generic (PLEG): container finished" podID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerID="8ba669bd84c81a3c1d3c0b8726badb26253b432a54c386034e16b8aa5de742fd" exitCode=60 Oct 31 21:30:19.668932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.668775 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" event={"ID":"24a8cf4f-0e49-40c9-bcae-9272cb45a364","Type":"ContainerDied","Data":"8ba669bd84c81a3c1d3c0b8726badb26253b432a54c386034e16b8aa5de742fd"} Oct 31 21:30:19.668932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.668838 5188 scope.go:122] "RemoveContainer" containerID="cb568de638b51aec4aabba795d02111c841527e6f984e58b1db76fbae26bbd7e" Oct 31 21:30:19.669283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.669255 5188 scope.go:122] "RemoveContainer" containerID="8ba669bd84c81a3c1d3c0b8726badb26253b432a54c386034e16b8aa5de742fd" Oct 31 21:30:19.669602 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:19.669582 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j_odh-ai-gateway-infra(24a8cf4f-0e49-40c9-bcae-9272cb45a364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" Oct 31 21:30:19.671200 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.671180 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" event={"ID":"913eb467-04c5-41f0-a26f-8e8739eb0277","Type":"ContainerStarted","Data":"9def1e714baa3a3cd5719778b1d73345274efe268dd0b38e07024db5862933af"} Oct 31 21:30:19.699849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.699015 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" event={"ID":"b718d7b4-9366-46b2-88a4-41dddfe3c7a6","Type":"ContainerStarted","Data":"37ad23c2c86a6740965289d4ee21b2e9928c6eab4f419f1bd68c5c166e8b6810"} Oct 31 21:30:19.751977 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:19.751854 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j"] Oct 31 21:30:20.064521 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.064425 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:30:20.064521 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.064486 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 21:30:20.714770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.714735 5188 generic.go:356] "Generic (PLEG): container finished" podID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerID="9def1e714baa3a3cd5719778b1d73345274efe268dd0b38e07024db5862933af" exitCode=60 Oct 31 21:30:20.715112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.714895 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" event={"ID":"913eb467-04c5-41f0-a26f-8e8739eb0277","Type":"ContainerDied","Data":"9def1e714baa3a3cd5719778b1d73345274efe268dd0b38e07024db5862933af"} Oct 31 21:30:20.715112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.714953 5188 scope.go:122] "RemoveContainer" containerID="e917dd2f944a8330bb22b906798ce5c78e5af490183c599ea88c475452e7d4b1" Oct 31 21:30:20.715380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.715363 5188 scope.go:122] "RemoveContainer" containerID="9def1e714baa3a3cd5719778b1d73345274efe268dd0b38e07024db5862933af" Oct 31 21:30:20.715726 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:20.715688 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891370-fb42f_odh-ai-gateway-infra(913eb467-04c5-41f0-a26f-8e8739eb0277)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" Oct 31 21:30:20.717487 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.717462 5188 generic.go:356] "Generic (PLEG): container finished" podID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerID="c689b14729121c1d5f31ed9007a2d1e78fd1f4093a3141707c9bac2248c619d6" exitCode=60 Oct 31 21:30:20.717560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.717530 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" event={"ID":"9593b80e-95b7-4454-ba3f-f7828865fbc9","Type":"ContainerDied","Data":"c689b14729121c1d5f31ed9007a2d1e78fd1f4093a3141707c9bac2248c619d6"} Oct 31 21:30:20.717956 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.717936 5188 scope.go:122] "RemoveContainer" containerID="c689b14729121c1d5f31ed9007a2d1e78fd1f4093a3141707c9bac2248c619d6" Oct 31 21:30:20.718263 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:20.718229 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s_odh-ai-gateway-infra(9593b80e-95b7-4454-ba3f-f7828865fbc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" Oct 31 21:30:20.719931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.719905 5188 generic.go:356] "Generic (PLEG): container finished" podID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerID="37ad23c2c86a6740965289d4ee21b2e9928c6eab4f419f1bd68c5c166e8b6810" exitCode=60 Oct 31 21:30:20.720053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.719948 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" event={"ID":"b718d7b4-9366-46b2-88a4-41dddfe3c7a6","Type":"ContainerDied","Data":"37ad23c2c86a6740965289d4ee21b2e9928c6eab4f419f1bd68c5c166e8b6810"} Oct 31 21:30:20.720352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.720303 5188 scope.go:122] "RemoveContainer" containerID="37ad23c2c86a6740965289d4ee21b2e9928c6eab4f419f1bd68c5c166e8b6810" Oct 31 21:30:20.720658 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:20.720635 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v_odh-ai-gateway-infra(b718d7b4-9366-46b2-88a4-41dddfe3c7a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" Oct 31 21:30:20.729805 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.729774 5188 scope.go:122] "RemoveContainer" containerID="ad9f57d85ef70bd4bf2d2435814fb8582bbee206ee55c68e36d89bfc7dd564df" Oct 31 21:30:20.748056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.746755 5188 scope.go:122] "RemoveContainer" containerID="7ada70aa0ff5016bef1aa73eb0d431af02528f7fb3bad04f15162ada29ee498f" Oct 31 21:30:20.796956 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.796897 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f"] Oct 31 21:30:20.850377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.850337 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v"] Oct 31 21:30:20.914368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.913474 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:20.967926 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.967836 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a8cf4f-0e49-40c9-bcae-9272cb45a364-kube-api-access-rl4fg\" (UniqueName: \"kubernetes.io/projected/24a8cf4f-0e49-40c9-bcae-9272cb45a364-kube-api-access-rl4fg\") pod \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\" (UID: \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\") " Oct 31 21:30:20.968160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.968137 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a8cf4f-0e49-40c9-bcae-9272cb45a364-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a8cf4f-0e49-40c9-bcae-9272cb45a364-maas-api-service-ca\") pod \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\" (UID: \"24a8cf4f-0e49-40c9-bcae-9272cb45a364\") " Oct 31 21:30:20.969123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.969094 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a8cf4f-0e49-40c9-bcae-9272cb45a364-maas-api-service-ca" pod "24a8cf4f-0e49-40c9-bcae-9272cb45a364" (UID: "24a8cf4f-0e49-40c9-bcae-9272cb45a364"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:20.985485 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:20.985447 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a8cf4f-0e49-40c9-bcae-9272cb45a364-kube-api-access-rl4fg" pod "24a8cf4f-0e49-40c9-bcae-9272cb45a364" (UID: "24a8cf4f-0e49-40c9-bcae-9272cb45a364"). InnerVolumeSpecName "kube-api-access-rl4fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:21.069774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.069730 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a8cf4f-0e49-40c9-bcae-9272cb45a364-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:21.069774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.069776 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl4fg\" (UniqueName: \"kubernetes.io/projected/24a8cf4f-0e49-40c9-bcae-9272cb45a364-kube-api-access-rl4fg\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:21.385669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.385631 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:21.387410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.387381 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:21.388830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.388802 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 21:30:21.742394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.737987 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" event={"ID":"24a8cf4f-0e49-40c9-bcae-9272cb45a364","Type":"ContainerDied","Data":"f9011c74d12e7f7fc91247095b9b7e785737dfc71f11220616391672f0752b26"} Oct 31 21:30:21.742394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.738067 5188 scope.go:122] "RemoveContainer" containerID="8ba669bd84c81a3c1d3c0b8726badb26253b432a54c386034e16b8aa5de742fd" Oct 31 21:30:21.742394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.738206 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j" Oct 31 21:30:21.818572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.818530 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j"] Oct 31 21:30:21.829919 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.829875 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s"] Oct 31 21:30:21.834644 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:21.833197 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891370-4jp9j"] Oct 31 21:30:22.184288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.183790 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:22.193664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.193580 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:22.335174 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.335136 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-maas-api-service-ca\") pod \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\" (UID: \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\") " Oct 31 21:30:22.335588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.335554 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913eb467-04c5-41f0-a26f-8e8739eb0277-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913eb467-04c5-41f0-a26f-8e8739eb0277-maas-api-service-ca\") pod \"913eb467-04c5-41f0-a26f-8e8739eb0277\" (UID: \"913eb467-04c5-41f0-a26f-8e8739eb0277\") " Oct 31 21:30:22.336372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.336285 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-kube-api-access-2dch8\" (UniqueName: \"kubernetes.io/projected/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-kube-api-access-2dch8\") pod \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\" (UID: \"b718d7b4-9366-46b2-88a4-41dddfe3c7a6\") " Oct 31 21:30:22.336612 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.336552 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913eb467-04c5-41f0-a26f-8e8739eb0277-kube-api-access-kqvv9\" (UniqueName: \"kubernetes.io/projected/913eb467-04c5-41f0-a26f-8e8739eb0277-kube-api-access-kqvv9\") pod \"913eb467-04c5-41f0-a26f-8e8739eb0277\" (UID: \"913eb467-04c5-41f0-a26f-8e8739eb0277\") " Oct 31 21:30:22.338849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.338815 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:22.340383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.340276 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-maas-api-service-ca" pod "b718d7b4-9366-46b2-88a4-41dddfe3c7a6" (UID: "b718d7b4-9366-46b2-88a4-41dddfe3c7a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:22.346545 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.346507 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913eb467-04c5-41f0-a26f-8e8739eb0277-maas-api-service-ca" pod "913eb467-04c5-41f0-a26f-8e8739eb0277" (UID: "913eb467-04c5-41f0-a26f-8e8739eb0277"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:22.393783 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.393658 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-kube-api-access-2dch8" pod "b718d7b4-9366-46b2-88a4-41dddfe3c7a6" (UID: "b718d7b4-9366-46b2-88a4-41dddfe3c7a6"). InnerVolumeSpecName "kube-api-access-2dch8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:22.394815 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.394644 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913eb467-04c5-41f0-a26f-8e8739eb0277-kube-api-access-kqvv9" pod "913eb467-04c5-41f0-a26f-8e8739eb0277" (UID: "913eb467-04c5-41f0-a26f-8e8739eb0277"). InnerVolumeSpecName "kube-api-access-kqvv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:22.439680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.438587 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9593b80e-95b7-4454-ba3f-f7828865fbc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9593b80e-95b7-4454-ba3f-f7828865fbc9-maas-api-service-ca\") pod \"9593b80e-95b7-4454-ba3f-f7828865fbc9\" (UID: \"9593b80e-95b7-4454-ba3f-f7828865fbc9\") " Oct 31 21:30:22.439680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.438795 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9593b80e-95b7-4454-ba3f-f7828865fbc9-kube-api-access-9fr4h\" (UniqueName: \"kubernetes.io/projected/9593b80e-95b7-4454-ba3f-f7828865fbc9-kube-api-access-9fr4h\") pod \"9593b80e-95b7-4454-ba3f-f7828865fbc9\" (UID: \"9593b80e-95b7-4454-ba3f-f7828865fbc9\") " Oct 31 21:30:22.439680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.439190 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:22.439680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.439217 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/913eb467-04c5-41f0-a26f-8e8739eb0277-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:22.439680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.439234 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dch8\" (UniqueName: \"kubernetes.io/projected/b718d7b4-9366-46b2-88a4-41dddfe3c7a6-kube-api-access-2dch8\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:22.439680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.439247 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqvv9\" (UniqueName: \"kubernetes.io/projected/913eb467-04c5-41f0-a26f-8e8739eb0277-kube-api-access-kqvv9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:22.440599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.440571 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9593b80e-95b7-4454-ba3f-f7828865fbc9-maas-api-service-ca" pod "9593b80e-95b7-4454-ba3f-f7828865fbc9" (UID: "9593b80e-95b7-4454-ba3f-f7828865fbc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:22.442105 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.442062 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9593b80e-95b7-4454-ba3f-f7828865fbc9-kube-api-access-9fr4h" pod "9593b80e-95b7-4454-ba3f-f7828865fbc9" (UID: "9593b80e-95b7-4454-ba3f-f7828865fbc9"). InnerVolumeSpecName "kube-api-access-9fr4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:22.541192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.541095 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fr4h\" (UniqueName: \"kubernetes.io/projected/9593b80e-95b7-4454-ba3f-f7828865fbc9-kube-api-access-9fr4h\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:22.541438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.541415 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9593b80e-95b7-4454-ba3f-f7828865fbc9-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:22.822686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.822587 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" event={"ID":"913eb467-04c5-41f0-a26f-8e8739eb0277","Type":"ContainerDied","Data":"da41b7a40f4095d16fa3364439175cf00cec9d1385718da51aad775a94b794b7"} Oct 31 21:30:22.823236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.823217 5188 scope.go:122] "RemoveContainer" containerID="9def1e714baa3a3cd5719778b1d73345274efe268dd0b38e07024db5862933af" Oct 31 21:30:22.823460 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.823445 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f" Oct 31 21:30:22.848074 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.847607 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" event={"ID":"9593b80e-95b7-4454-ba3f-f7828865fbc9","Type":"ContainerDied","Data":"74fb6b4724907b9ef24568e2bbed162d3635b24e368abc0f584b948a6b8ee056"} Oct 31 21:30:22.849414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.848430 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s" Oct 31 21:30:22.852815 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.852795 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" Oct 31 21:30:22.853642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.853416 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v" event={"ID":"b718d7b4-9366-46b2-88a4-41dddfe3c7a6","Type":"ContainerDied","Data":"15953d5f953c295ac93af77383eb46603222fb1aab857557b7bb1ebdf1b27c65"} Oct 31 21:30:23.000424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.992477 5188 scope.go:122] "RemoveContainer" containerID="c689b14729121c1d5f31ed9007a2d1e78fd1f4093a3141707c9bac2248c619d6" Oct 31 21:30:23.000424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:22.993455 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f"] Oct 31 21:30:23.080496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.065426 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891370-fb42f"] Oct 31 21:30:23.080496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.075624 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s"] Oct 31 21:30:23.122149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.122076 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891370-kd28s"] Oct 31 21:30:23.133419 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.133381 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v"] Oct 31 21:30:23.166978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.166947 5188 scope.go:122] "RemoveContainer" containerID="37ad23c2c86a6740965289d4ee21b2e9928c6eab4f419f1bd68c5c166e8b6810" Oct 31 21:30:23.176244 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.176173 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891370-h6j4v"] Oct 31 21:30:23.738350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.718598 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" path="/var/lib/kubelet/pods/24a8cf4f-0e49-40c9-bcae-9272cb45a364/volumes" Oct 31 21:30:23.742984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.742947 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" path="/var/lib/kubelet/pods/913eb467-04c5-41f0-a26f-8e8739eb0277/volumes" Oct 31 21:30:23.745547 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.743893 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" path="/var/lib/kubelet/pods/9593b80e-95b7-4454-ba3f-f7828865fbc9/volumes" Oct 31 21:30:23.818347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:23.795168 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" path="/var/lib/kubelet/pods/b718d7b4-9366-46b2-88a4-41dddfe3c7a6/volumes" Oct 31 21:30:24.228349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:24.227414 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:24.305348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:24.304421 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:25.635412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:25.635380 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:30:25.718722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:25.718686 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc"] Oct 31 21:30:27.604968 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.604932 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97"] Oct 31 21:30:27.605552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605534 5188 cpu_manager.go:414] "containerMap: removing container" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerName="cleanup" Oct 31 21:30:27.605552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605552 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605572 5188 cpu_manager.go:414] "containerMap: removing container" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605576 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605589 5188 cpu_manager.go:414] "containerMap: removing container" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605594 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605602 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605607 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605613 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605617 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605629 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605634 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605640 5188 cpu_manager.go:414] "containerMap: removing container" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605644 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605651 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605655 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerName="cleanup" Oct 31 21:30:27.605669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605677 5188 cpu_manager.go:414] "containerMap: removing container" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605682 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605765 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605773 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605780 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605786 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605792 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605798 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605807 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605813 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605818 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerName="cleanup" Oct 31 21:30:27.606382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.605824 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerName="cleanup" Oct 31 21:30:27.611151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.611128 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.639602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.639537 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh"] Oct 31 21:30:27.641104 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.641068 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerName="cleanup" Oct 31 21:30:27.641104 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.641101 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerName="cleanup" Oct 31 21:30:27.641309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.641123 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerName="cleanup" Oct 31 21:30:27.641309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.641131 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b718d7b4-9366-46b2-88a4-41dddfe3c7a6" containerName="cleanup" Oct 31 21:30:27.641309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.641264 5188 cpu_manager.go:414] "containerMap: removing container" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerName="cleanup" Oct 31 21:30:27.641309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.641274 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a8cf4f-0e49-40c9-bcae-9272cb45a364" containerName="cleanup" Oct 31 21:30:27.641579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.641564 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9593b80e-95b7-4454-ba3f-f7828865fbc9" containerName="cleanup" Oct 31 21:30:27.641579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.641582 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="913eb467-04c5-41f0-a26f-8e8739eb0277" containerName="cleanup" Oct 31 21:30:27.647242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.647216 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97"] Oct 31 21:30:27.647408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.647316 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.665922 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.665890 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh"] Oct 31 21:30:27.673943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.673909 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420fbd0f-3194-49cb-9a43-191992684189-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.674073 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.673971 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.674073 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.674035 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.674179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.674071 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.674179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.674115 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhbl\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-kube-api-access-hjhbl\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.674179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.674145 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420fbd0f-3194-49cb-9a43-191992684189-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.674179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.674172 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.674304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.674226 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.674304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.674287 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.775498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775454 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.775498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775503 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420fbd0f-3194-49cb-9a43-191992684189-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.775770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775520 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.775770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775665 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.775770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775721 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.775932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775792 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.775932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775831 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.775932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775861 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.775932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775891 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.775932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775917 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.776138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775947 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sswbb\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-kube-api-access-sswbb\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.776138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.775994 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhbl\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-kube-api-access-hjhbl\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.776138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776055 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.776287 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776136 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.776287 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776215 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420fbd0f-3194-49cb-9a43-191992684189-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.776287 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776259 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.776466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776307 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.776466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776367 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.776466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776380 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420fbd0f-3194-49cb-9a43-191992684189-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.776580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776496 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.776618 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776592 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.776950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.776926 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.777059 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.777010 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.778762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.778738 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.778914 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.778883 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420fbd0f-3194-49cb-9a43-191992684189-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.808993 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.808921 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.810633 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.810604 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhbl\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-kube-api-access-hjhbl\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.877363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877308 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.877363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877364 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.877619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877417 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.877619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877467 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.877619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877494 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.877619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877515 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.877619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877539 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sswbb\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-kube-api-access-sswbb\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.877926 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877627 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.877926 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877664 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.877926 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.877806 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.878230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.878206 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.878319 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.878240 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.878436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.878417 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.878559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.878533 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.880481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.880457 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.880573 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.880484 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.899082 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.899055 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.899205 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.899173 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sswbb\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-kube-api-access-sswbb\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:27.927011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.926972 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:27.972010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:27.970770 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:28.253382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:28.253349 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97"] Oct 31 21:30:28.254166 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:28.254129 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420fbd0f_3194_49cb_9a43_191992684189.slice/crio-89d3a95898452abc320dd7b9d45e31c99696ffafc6e2e56ede415a8ecc8b9413 WatchSource:0}: Error finding container 89d3a95898452abc320dd7b9d45e31c99696ffafc6e2e56ede415a8ecc8b9413: Status 404 returned error can't find the container with id 89d3a95898452abc320dd7b9d45e31c99696ffafc6e2e56ede415a8ecc8b9413 Oct 31 21:30:28.256864 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:28.256828 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:28.256981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:28.256906 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:28.256981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:28.256949 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:28.315453 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:28.315414 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4bd232_cac5_4ab4_aedb_5242b6e59f81.slice/crio-4fbef5455c5fadc9b106e2ecba28c3bb2c1fe16bd6430a327ffcfbcc6c4f89c7 WatchSource:0}: Error finding container 4fbef5455c5fadc9b106e2ecba28c3bb2c1fe16bd6430a327ffcfbcc6c4f89c7: Status 404 returned error can't find the container with id 4fbef5455c5fadc9b106e2ecba28c3bb2c1fe16bd6430a327ffcfbcc6c4f89c7 Oct 31 21:30:28.316259 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:28.316232 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh"] Oct 31 21:30:28.318445 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:28.318405 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:28.318553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:28.318494 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:28.318553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:28.318538 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:29.035400 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.035359 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" event={"ID":"420fbd0f-3194-49cb-9a43-191992684189","Type":"ContainerStarted","Data":"8c5a2e32ab7fecce2f0c77b7e138903284415a378fe287efdde519143890a025"} Oct 31 21:30:29.035880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.035427 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" event={"ID":"420fbd0f-3194-49cb-9a43-191992684189","Type":"ContainerStarted","Data":"89d3a95898452abc320dd7b9d45e31c99696ffafc6e2e56ede415a8ecc8b9413"} Oct 31 21:30:29.037374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.037346 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" event={"ID":"0f4bd232-cac5-4ab4-aedb-5242b6e59f81","Type":"ContainerStarted","Data":"d05cd36aa43ad319cbfc62c76cf8f88fba35885bddfe6db359012049ea262f9c"} Oct 31 21:30:29.037498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.037400 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" event={"ID":"0f4bd232-cac5-4ab4-aedb-5242b6e59f81","Type":"ContainerStarted","Data":"4fbef5455c5fadc9b106e2ecba28c3bb2c1fe16bd6430a327ffcfbcc6c4f89c7"} Oct 31 21:30:29.093902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.093825 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podStartSLOduration=2.093806601 podStartE2EDuration="2.093806601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:29.088317843 +0000 UTC m=+875.781275235" watchObservedRunningTime="2026-10-31 21:30:29.093806601 +0000 UTC m=+875.786763994" Oct 31 21:30:29.175360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.173828 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podStartSLOduration=2.173805485 podStartE2EDuration="2.173805485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:29.171640156 +0000 UTC m=+875.864597549" watchObservedRunningTime="2026-10-31 21:30:29.173805485 +0000 UTC m=+875.866762878" Oct 31 21:30:29.930347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.929778 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:29.934538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.934497 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 21:30:29.934825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.934781 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 21:30:29.974507 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.974415 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:29.992897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.992523 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 21:30:29.992897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:29.992616 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 21:30:30.719453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:30.719398 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" podUID="c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" containerName="payload-processing" containerID="cri-o://ad20a79f870def77dd0bf54a1068a328ff397b9357b6f8d029f34c37eef0d148" gracePeriod=25 Oct 31 21:30:30.938801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:30.938367 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 21:30:30.938801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:30.938417 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 21:30:30.980417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:30.977489 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 21:30:30.980417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:30.977563 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 21:30:31.061242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.060475 5188 generic.go:356] "Generic (PLEG): container finished" podID="c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" containerID="ad20a79f870def77dd0bf54a1068a328ff397b9357b6f8d029f34c37eef0d148" exitCode=0 Oct 31 21:30:31.061242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.060989 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" event={"ID":"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af","Type":"ContainerDied","Data":"ad20a79f870def77dd0bf54a1068a328ff397b9357b6f8d029f34c37eef0d148"} Oct 31 21:30:31.243064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.242988 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:30:31.370225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.368600 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-provider-credentials\") pod \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " Oct 31 21:30:31.370225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.368661 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-routing-overlay-volume\") pod \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " Oct 31 21:30:31.370225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.368820 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-kube-api-access-kht4h\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-kube-api-access-kht4h\") pod \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " Oct 31 21:30:31.370225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.368866 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-plugins-config-volume\") pod \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\" (UID: \"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af\") " Oct 31 21:30:31.370225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.369966 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-plugins-config-volume" pod "c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" (UID: "c784bd92-84dd-489a-9b6d-5ef6b5dcf5af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:31.417609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.392488 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-kube-api-access-kht4h" pod "c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" (UID: "c784bd92-84dd-489a-9b6d-5ef6b5dcf5af"). InnerVolumeSpecName "kube-api-access-kht4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:31.417609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.392705 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-routing-overlay-volume" pod "c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" (UID: "c784bd92-84dd-489a-9b6d-5ef6b5dcf5af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:31.440758 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.440684 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-provider-credentials" pod "c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" (UID: "c784bd92-84dd-489a-9b6d-5ef6b5dcf5af"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:31.470428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.470394 5188 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-provider-credentials\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:31.470428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.470422 5188 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-routing-overlay-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:31.470428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.470431 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kht4h\" (UniqueName: \"kubernetes.io/projected/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-kube-api-access-kht4h\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:31.470428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.470440 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:31.939349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.938970 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 21:30:31.939349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.939032 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 21:30:31.977349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.975917 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 21:30:31.977349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:31.976010 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 21:30:32.207584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:32.207478 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" event={"ID":"c784bd92-84dd-489a-9b6d-5ef6b5dcf5af","Type":"ContainerDied","Data":"8ba194fe5b8c4bd817266784766d6b003374aec0691cae2880117672226a7ddc"} Oct 31 21:30:32.207857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:32.207836 5188 scope.go:122] "RemoveContainer" containerID="ad20a79f870def77dd0bf54a1068a328ff397b9357b6f8d029f34c37eef0d148" Oct 31 21:30:32.208151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:32.208131 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc" Oct 31 21:30:32.307093 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:32.307048 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc"] Oct 31 21:30:32.322744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:32.321851 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-9888fbfd6-ggrmc"] Oct 31 21:30:32.831348 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:30:32.817731 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:30:32.930346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:32.929745 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 21:30:32.930346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:32.929809 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 21:30:32.977345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:32.976841 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 21:30:32.977345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:32.976900 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 21:30:33.262362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:33.258907 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:33.269499 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:33.269450 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:33.553195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:33.552358 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" path="/var/lib/kubelet/pods/c784bd92-84dd-489a-9b6d-5ef6b5dcf5af/volumes" Oct 31 21:30:33.938423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:33.937944 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 21:30:33.938423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:33.938006 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 21:30:33.999071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:33.999038 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 21:30:33.999709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:33.999674 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 21:30:34.930939 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:34.930882 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 21:30:34.931268 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:34.931236 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 21:30:34.972853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:34.972796 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 21:30:34.973149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:34.973114 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 21:30:35.408487 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:35.408427 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:35.688407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:35.663481 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:35.929785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:35.929743 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 21:30:35.930094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:35.930060 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 21:30:35.971848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:35.971751 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 21:30:35.972124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:35.972091 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 21:30:36.502349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:36.455453 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:36.539350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:36.518881 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:30:36.860528 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:36.860491 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:30:36.951234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:36.930266 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 31 21:30:36.955068 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:36.955011 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 31 21:30:37.022387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:36.988573 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 21:30:37.022750 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:37.022705 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 21:30:37.976851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:37.976811 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 31 21:30:37.977517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:37.977467 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 31 21:30:38.899484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:38.899453 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:38.899950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:38.899930 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:38.900734 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:38.900715 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:30:39.020255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:39.020220 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:39.020782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:39.020643 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:39.021974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:39.021949 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:30:43.828499 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.828458 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv"] Oct 31 21:30:43.829059 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.829045 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" containerName="payload-processing" Oct 31 21:30:43.829102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.829060 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" containerName="payload-processing" Oct 31 21:30:43.829151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.829142 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c784bd92-84dd-489a-9b6d-5ef6b5dcf5af" containerName="payload-processing" Oct 31 21:30:43.832606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.832587 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:43.853955 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.853922 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:30:43.857551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.857522 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:30:43.857721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.857601 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:30:43.857721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.857663 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:30:43.858008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.857967 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv"] Oct 31 21:30:43.900075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.900030 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f0f65e0-x7jxv\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:43.900075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.900074 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjzph\" (UniqueName: \"kubernetes.io/projected/fa255a70-9477-4694-9f01-df562b2dbead-kube-api-access-vjzph\") pod \"maas-api-revoke-sub-2f0f65e0-x7jxv\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:43.900372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:43.900192 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f0f65e0-x7jxv\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:44.002018 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:44.001966 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f0f65e0-x7jxv\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:44.002196 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:44.002164 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f0f65e0-x7jxv\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:44.002271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:44.002201 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjzph\" (UniqueName: \"kubernetes.io/projected/fa255a70-9477-4694-9f01-df562b2dbead-kube-api-access-vjzph\") pod \"maas-api-revoke-sub-2f0f65e0-x7jxv\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:44.002995 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:44.002966 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f0f65e0-x7jxv\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:44.004611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:44.004580 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f0f65e0-x7jxv\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:44.024265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:44.024231 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjzph\" (UniqueName: \"kubernetes.io/projected/fa255a70-9477-4694-9f01-df562b2dbead-kube-api-access-vjzph\") pod \"maas-api-revoke-sub-2f0f65e0-x7jxv\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:44.147619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:44.147581 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:44.461225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:44.461187 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv"] Oct 31 21:30:44.461374 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:44.461338 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa255a70_9477_4694_9f01_df562b2dbead.slice/crio-ea59e2c6b2c06ce999f9e282669770b9ceb7db5c54c8745ff94c43833419e468 WatchSource:0}: Error finding container ea59e2c6b2c06ce999f9e282669770b9ceb7db5c54c8745ff94c43833419e468: Status 404 returned error can't find the container with id ea59e2c6b2c06ce999f9e282669770b9ceb7db5c54c8745ff94c43833419e468 Oct 31 21:30:44.614002 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:44.613964 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" event={"ID":"fa255a70-9477-4694-9f01-df562b2dbead","Type":"ContainerStarted","Data":"ea59e2c6b2c06ce999f9e282669770b9ceb7db5c54c8745ff94c43833419e468"} Oct 31 21:30:45.625357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:45.625309 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" event={"ID":"fa255a70-9477-4694-9f01-df562b2dbead","Type":"ContainerStarted","Data":"2d186d0683cfe09c2023d4698937085838ade80bfa6f1247484b05187faa43d6"} Oct 31 21:30:45.661665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:45.661614 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" podStartSLOduration=2.661599064 podStartE2EDuration="2.661599064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:45.660926868 +0000 UTC m=+892.353884261" watchObservedRunningTime="2026-10-31 21:30:45.661599064 +0000 UTC m=+892.354556454" Oct 31 21:30:46.635803 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:46.635764 5188 generic.go:356] "Generic (PLEG): container finished" podID="fa255a70-9477-4694-9f01-df562b2dbead" containerID="2d186d0683cfe09c2023d4698937085838ade80bfa6f1247484b05187faa43d6" exitCode=0 Oct 31 21:30:46.636262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:46.635853 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" event={"ID":"fa255a70-9477-4694-9f01-df562b2dbead","Type":"ContainerDied","Data":"2d186d0683cfe09c2023d4698937085838ade80bfa6f1247484b05187faa43d6"} Oct 31 21:30:47.888680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:47.888638 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:47.963432 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:47.963301 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-xks-ca\") pod \"fa255a70-9477-4694-9f01-df562b2dbead\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " Oct 31 21:30:47.963432 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:47.963398 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa255a70-9477-4694-9f01-df562b2dbead-kube-api-access-vjzph\" (UniqueName: \"kubernetes.io/projected/fa255a70-9477-4694-9f01-df562b2dbead-kube-api-access-vjzph\") pod \"fa255a70-9477-4694-9f01-df562b2dbead\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " Oct 31 21:30:47.963706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:47.963537 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-openshift-ca\") pod \"fa255a70-9477-4694-9f01-df562b2dbead\" (UID: \"fa255a70-9477-4694-9f01-df562b2dbead\") " Oct 31 21:30:47.964208 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:47.964177 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-openshift-ca" pod "fa255a70-9477-4694-9f01-df562b2dbead" (UID: "fa255a70-9477-4694-9f01-df562b2dbead"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:47.970382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:47.966543 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-xks-ca" pod "fa255a70-9477-4694-9f01-df562b2dbead" (UID: "fa255a70-9477-4694-9f01-df562b2dbead"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:30:47.970382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:47.967858 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa255a70-9477-4694-9f01-df562b2dbead-kube-api-access-vjzph" pod "fa255a70-9477-4694-9f01-df562b2dbead" (UID: "fa255a70-9477-4694-9f01-df562b2dbead"). InnerVolumeSpecName "kube-api-access-vjzph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:48.065493 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:48.065444 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:48.065493 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:48.065487 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjzph\" (UniqueName: \"kubernetes.io/projected/fa255a70-9477-4694-9f01-df562b2dbead-kube-api-access-vjzph\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:48.065762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:48.065504 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa255a70-9477-4694-9f01-df562b2dbead-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:30:48.687344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:48.683738 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" event={"ID":"fa255a70-9477-4694-9f01-df562b2dbead","Type":"ContainerDied","Data":"ea59e2c6b2c06ce999f9e282669770b9ceb7db5c54c8745ff94c43833419e468"} Oct 31 21:30:48.687344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:48.683809 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea59e2c6b2c06ce999f9e282669770b9ceb7db5c54c8745ff94c43833419e468" Oct 31 21:30:48.687344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:48.683886 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv" Oct 31 21:30:53.660067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:53.660031 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:30:53.669614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:53.669583 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:30:56.184660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.184613 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf"] Oct 31 21:30:56.185478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.185391 5188 cpu_manager.go:414] "containerMap: removing container" podUID="fa255a70-9477-4694-9f01-df562b2dbead" containerName="revoke-keys" Oct 31 21:30:56.185478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.185410 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa255a70-9477-4694-9f01-df562b2dbead" containerName="revoke-keys" Oct 31 21:30:56.185642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.185558 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa255a70-9477-4694-9f01-df562b2dbead" containerName="revoke-keys" Oct 31 21:30:56.191177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.191155 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.203667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.203625 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:30:56.209042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.209008 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf"] Oct 31 21:30:56.213554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.213525 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:30:56.213698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.213646 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:30:56.214049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.214029 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:30:56.305313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.305274 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-912f1738-fgnhf\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.305313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.305336 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvj4g\" (UniqueName: \"kubernetes.io/projected/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-kube-api-access-kvj4g\") pod \"maas-api-revoke-sub-912f1738-fgnhf\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.305634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.305402 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-912f1738-fgnhf\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.413400 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.413363 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-912f1738-fgnhf\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.413693 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.413670 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvj4g\" (UniqueName: \"kubernetes.io/projected/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-kube-api-access-kvj4g\") pod \"maas-api-revoke-sub-912f1738-fgnhf\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.413893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.413875 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-912f1738-fgnhf\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.451466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.451351 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz"] Oct 31 21:30:56.456994 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.456945 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-912f1738-fgnhf\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.487641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.469235 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-912f1738-fgnhf\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.516978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.516932 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvj4g\" (UniqueName: \"kubernetes.io/projected/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-kube-api-access-kvj4g\") pod \"maas-api-revoke-sub-912f1738-fgnhf\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.574348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.573590 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz"] Oct 31 21:30:56.574348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.573719 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.744407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.744002 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b287x\" (UniqueName: \"kubernetes.io/projected/371aa73a-ad00-45eb-8216-d8f9116e4577-kube-api-access-b287x\") pod \"maas-api-revoke-sub-4f40e774-lpqkz\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.744407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.744066 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f40e774-lpqkz\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.744407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.744177 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f40e774-lpqkz\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.811349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.810544 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:30:56.849347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.849232 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b287x\" (UniqueName: \"kubernetes.io/projected/371aa73a-ad00-45eb-8216-d8f9116e4577-kube-api-access-b287x\") pod \"maas-api-revoke-sub-4f40e774-lpqkz\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.849347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.849297 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f40e774-lpqkz\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.852372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.849804 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f40e774-lpqkz\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.852372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.850782 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f40e774-lpqkz\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.869670 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.866776 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f40e774-lpqkz\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.876168 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.876132 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b287x\" (UniqueName: \"kubernetes.io/projected/371aa73a-ad00-45eb-8216-d8f9116e4577-kube-api-access-b287x\") pod \"maas-api-revoke-sub-4f40e774-lpqkz\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:56.954754 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:56.954473 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:30:57.695139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:57.695097 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf"] Oct 31 21:30:57.763024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:57.762991 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" event={"ID":"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b","Type":"ContainerStarted","Data":"69ee0006f711714fd3df53e2e6b8341ded732764609a86871b6100f5093176b5"} Oct 31 21:30:57.852243 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:57.852202 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz"] Oct 31 21:30:58.772349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.772272 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" event={"ID":"371aa73a-ad00-45eb-8216-d8f9116e4577","Type":"ContainerStarted","Data":"b1ce2548e8b090259e466a2de591921c12aad51abee2732f43af9be9ed2ccd7a"} Oct 31 21:30:58.772871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.772841 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" event={"ID":"371aa73a-ad00-45eb-8216-d8f9116e4577","Type":"ContainerStarted","Data":"b40aacb1c2de17a270c13575af17fa27b7bf6f4835c0131932173f3763589f30"} Oct 31 21:30:58.783086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.783044 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" event={"ID":"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b","Type":"ContainerStarted","Data":"d2ecc52f9c0bc3a56a860297ef9e3686c62ab8fb9e52adf7fcc670f547efa614"} Oct 31 21:30:58.811055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.810983 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" podStartSLOduration=2.810966032 podStartE2EDuration="2.810966032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:58.807777433 +0000 UTC m=+905.500734823" watchObservedRunningTime="2026-10-31 21:30:58.810966032 +0000 UTC m=+905.503923425" Oct 31 21:30:58.850650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.850596 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" podStartSLOduration=2.85058045 podStartE2EDuration="2.85058045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:58.847179181 +0000 UTC m=+905.540136577" watchObservedRunningTime="2026-10-31 21:30:58.85058045 +0000 UTC m=+905.543537840" Oct 31 21:30:58.905022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.904975 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p"] Oct 31 21:30:58.909585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.909544 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:58.941681 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.941642 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p"] Oct 31 21:30:58.978962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.978206 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:58.978962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.978302 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:58.978962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.978393 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fffft\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-kube-api-access-fffft\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:58.978962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.978426 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:58.978962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.978525 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:58.978962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.978552 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:58.978962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.978592 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:58.978962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.978698 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:58.978962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:58.978737 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.079477 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079366 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.079477 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079460 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.079741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079488 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.079741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079516 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.079741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079577 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.079741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079627 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fffft\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-kube-api-access-fffft\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.079741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079653 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.080509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079748 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.080509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079780 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.080509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.079946 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.080509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.080146 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.080509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.080220 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.080509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.080349 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.080844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.080816 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.082469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.082439 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.083080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.083056 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.102719 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.102683 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.103186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.103157 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fffft\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-kube-api-access-fffft\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.228647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.228607 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:30:59.550897 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:30:59.550849 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a17c3f_4dc9_4d95_9e63_1ed55bdb363b.slice/crio-cd057a13eb8c593ce385d6a9fe05302681d30a40b6331d42ca120ae48a991355 WatchSource:0}: Error finding container cd057a13eb8c593ce385d6a9fe05302681d30a40b6331d42ca120ae48a991355: Status 404 returned error can't find the container with id cd057a13eb8c593ce385d6a9fe05302681d30a40b6331d42ca120ae48a991355 Oct 31 21:30:59.551021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.550897 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p"] Oct 31 21:30:59.553425 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.553394 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:59.553511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.553483 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:59.553551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.553513 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:30:59.795934 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.795894 5188 generic.go:356] "Generic (PLEG): container finished" podID="371aa73a-ad00-45eb-8216-d8f9116e4577" containerID="b1ce2548e8b090259e466a2de591921c12aad51abee2732f43af9be9ed2ccd7a" exitCode=0 Oct 31 21:30:59.796412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.795982 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" event={"ID":"371aa73a-ad00-45eb-8216-d8f9116e4577","Type":"ContainerDied","Data":"b1ce2548e8b090259e466a2de591921c12aad51abee2732f43af9be9ed2ccd7a"} Oct 31 21:30:59.798066 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.797998 5188 generic.go:356] "Generic (PLEG): container finished" podID="b0c1703c-89ee-4fda-9b11-c705c3c2ad0b" containerID="d2ecc52f9c0bc3a56a860297ef9e3686c62ab8fb9e52adf7fcc670f547efa614" exitCode=0 Oct 31 21:30:59.798209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.798089 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" event={"ID":"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b","Type":"ContainerDied","Data":"d2ecc52f9c0bc3a56a860297ef9e3686c62ab8fb9e52adf7fcc670f547efa614"} Oct 31 21:30:59.800029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.800005 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" event={"ID":"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b","Type":"ContainerStarted","Data":"75349768e0454cb9378a9e43b76199ed45cf679a5c55460f81517d3c93f9d3fd"} Oct 31 21:30:59.800146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.800051 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" event={"ID":"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b","Type":"ContainerStarted","Data":"cd057a13eb8c593ce385d6a9fe05302681d30a40b6331d42ca120ae48a991355"} Oct 31 21:30:59.914160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:30:59.914100 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" podStartSLOduration=1.9140808150000002 podStartE2EDuration="1.914080815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:59.912862228 +0000 UTC m=+906.605819621" watchObservedRunningTime="2026-10-31 21:30:59.914080815 +0000 UTC m=+906.607038209" Oct 31 21:31:00.229646 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:00.229599 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:31:00.231551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:00.231487 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 21:31:00.231745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:00.231576 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 21:31:00.966384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:00.962766 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4"] Oct 31 21:31:00.976524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:00.970666 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.022350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.020414 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4"] Oct 31 21:31:01.022350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.021537 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9dadda-mvbm4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.022350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.021754 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9dadda-mvbm4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.022350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.021789 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whr6g\" (UniqueName: \"kubernetes.io/projected/17258272-6b47-43d4-929b-62e74e9b3ce4-kube-api-access-whr6g\") pod \"maas-api-revoke-sub-0e9dadda-mvbm4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.114851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.114805 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:31:01.118501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.118476 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:31:01.123195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.123169 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9dadda-mvbm4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.123255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.123217 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whr6g\" (UniqueName: \"kubernetes.io/projected/17258272-6b47-43d4-929b-62e74e9b3ce4-kube-api-access-whr6g\") pod \"maas-api-revoke-sub-0e9dadda-mvbm4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.123315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.123296 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9dadda-mvbm4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.124236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.124206 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e9dadda-mvbm4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.125992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.125923 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e9dadda-mvbm4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.145363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.145309 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whr6g\" (UniqueName: \"kubernetes.io/projected/17258272-6b47-43d4-929b-62e74e9b3ce4-kube-api-access-whr6g\") pod \"maas-api-revoke-sub-0e9dadda-mvbm4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.227173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.224672 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-openshift-ca\") pod \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " Oct 31 21:31:01.227173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.224727 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-kube-api-access-kvj4g\" (UniqueName: \"kubernetes.io/projected/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-kube-api-access-kvj4g\") pod \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " Oct 31 21:31:01.227173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.224764 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-openshift-ca\") pod \"371aa73a-ad00-45eb-8216-d8f9116e4577\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " Oct 31 21:31:01.227173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.224924 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371aa73a-ad00-45eb-8216-d8f9116e4577-kube-api-access-b287x\" (UniqueName: \"kubernetes.io/projected/371aa73a-ad00-45eb-8216-d8f9116e4577-kube-api-access-b287x\") pod \"371aa73a-ad00-45eb-8216-d8f9116e4577\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " Oct 31 21:31:01.227173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.225010 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-xks-ca\") pod \"371aa73a-ad00-45eb-8216-d8f9116e4577\" (UID: \"371aa73a-ad00-45eb-8216-d8f9116e4577\") " Oct 31 21:31:01.227173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.225068 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-xks-ca\") pod \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\" (UID: \"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b\") " Oct 31 21:31:01.227173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.226625 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-openshift-ca" pod "371aa73a-ad00-45eb-8216-d8f9116e4577" (UID: "371aa73a-ad00-45eb-8216-d8f9116e4577"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:01.227173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.226996 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-openshift-ca" pod "b0c1703c-89ee-4fda-9b11-c705c3c2ad0b" (UID: "b0c1703c-89ee-4fda-9b11-c705c3c2ad0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:01.228433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.228252 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-kube-api-access-kvj4g" pod "b0c1703c-89ee-4fda-9b11-c705c3c2ad0b" (UID: "b0c1703c-89ee-4fda-9b11-c705c3c2ad0b"). InnerVolumeSpecName "kube-api-access-kvj4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:01.229513 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.228937 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 21:31:01.229513 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.228988 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 21:31:01.229940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.229906 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-xks-ca" pod "371aa73a-ad00-45eb-8216-d8f9116e4577" (UID: "371aa73a-ad00-45eb-8216-d8f9116e4577"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:01.230897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.230874 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371aa73a-ad00-45eb-8216-d8f9116e4577-kube-api-access-b287x" pod "371aa73a-ad00-45eb-8216-d8f9116e4577" (UID: "371aa73a-ad00-45eb-8216-d8f9116e4577"). InnerVolumeSpecName "kube-api-access-b287x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:01.231626 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.231608 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-xks-ca" pod "b0c1703c-89ee-4fda-9b11-c705c3c2ad0b" (UID: "b0c1703c-89ee-4fda-9b11-c705c3c2ad0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:01.326807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.326711 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.326807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.326756 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.326807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.326769 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvj4g\" (UniqueName: \"kubernetes.io/projected/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b-kube-api-access-kvj4g\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.326807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.326783 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.326807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.326796 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b287x\" (UniqueName: \"kubernetes.io/projected/371aa73a-ad00-45eb-8216-d8f9116e4577-kube-api-access-b287x\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.326807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.326809 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371aa73a-ad00-45eb-8216-d8f9116e4577-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.327233 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.326935 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:01.654666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.654624 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4"] Oct 31 21:31:01.655384 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:01.655350 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17258272_6b47_43d4_929b_62e74e9b3ce4.slice/crio-871c992294eb9bb75ee595d4efa81d5356a8f0b7ec7f4e75c7db6c8f1dcf593b WatchSource:0}: Error finding container 871c992294eb9bb75ee595d4efa81d5356a8f0b7ec7f4e75c7db6c8f1dcf593b: Status 404 returned error can't find the container with id 871c992294eb9bb75ee595d4efa81d5356a8f0b7ec7f4e75c7db6c8f1dcf593b Oct 31 21:31:01.819020 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.818979 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" Oct 31 21:31:01.819020 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.819006 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf" event={"ID":"b0c1703c-89ee-4fda-9b11-c705c3c2ad0b","Type":"ContainerDied","Data":"69ee0006f711714fd3df53e2e6b8341ded732764609a86871b6100f5093176b5"} Oct 31 21:31:01.819246 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.819058 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69ee0006f711714fd3df53e2e6b8341ded732764609a86871b6100f5093176b5" Oct 31 21:31:01.820537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.820509 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" event={"ID":"17258272-6b47-43d4-929b-62e74e9b3ce4","Type":"ContainerStarted","Data":"871c992294eb9bb75ee595d4efa81d5356a8f0b7ec7f4e75c7db6c8f1dcf593b"} Oct 31 21:31:01.822098 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.822062 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" Oct 31 21:31:01.822098 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.822080 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz" event={"ID":"371aa73a-ad00-45eb-8216-d8f9116e4577","Type":"ContainerDied","Data":"b40aacb1c2de17a270c13575af17fa27b7bf6f4835c0131932173f3763589f30"} Oct 31 21:31:01.822286 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:01.822137 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b40aacb1c2de17a270c13575af17fa27b7bf6f4835c0131932173f3763589f30" Oct 31 21:31:02.229065 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:02.228970 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 21:31:02.229065 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:02.229032 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 21:31:02.540313 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:31:02.540219 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:31:02.831045 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:02.831007 5188 generic.go:356] "Generic (PLEG): container finished" podID="17258272-6b47-43d4-929b-62e74e9b3ce4" containerID="c8b27fe69b0db8b8f0058d5ce95028e6f16462495a150a1337429d46c271c535" exitCode=0 Oct 31 21:31:02.831242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:02.831091 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" event={"ID":"17258272-6b47-43d4-929b-62e74e9b3ce4","Type":"ContainerDied","Data":"c8b27fe69b0db8b8f0058d5ce95028e6f16462495a150a1337429d46c271c535"} Oct 31 21:31:03.157195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.157159 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c"] Oct 31 21:31:03.157768 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.157754 5188 cpu_manager.go:414] "containerMap: removing container" podUID="371aa73a-ad00-45eb-8216-d8f9116e4577" containerName="revoke-keys" Oct 31 21:31:03.157810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.157770 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371aa73a-ad00-45eb-8216-d8f9116e4577" containerName="revoke-keys" Oct 31 21:31:03.157849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.157813 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b0c1703c-89ee-4fda-9b11-c705c3c2ad0b" containerName="revoke-keys" Oct 31 21:31:03.157849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.157819 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c1703c-89ee-4fda-9b11-c705c3c2ad0b" containerName="revoke-keys" Oct 31 21:31:03.157913 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.157905 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c1703c-89ee-4fda-9b11-c705c3c2ad0b" containerName="revoke-keys" Oct 31 21:31:03.157943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.157915 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="371aa73a-ad00-45eb-8216-d8f9116e4577" containerName="revoke-keys" Oct 31 21:31:03.161495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.161473 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.219931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.219884 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c"] Oct 31 21:31:03.232343 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.229629 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 21:31:03.232343 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.229689 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 21:31:03.252017 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.251977 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.252235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.252026 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.252235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.252176 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.252403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.252254 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.252403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.252301 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.252403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.252361 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.252572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.252546 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.252697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.252678 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bf0314-53b2-4937-9106-f72080e6d0f1-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.252753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.252737 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj58w\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-kube-api-access-kj58w\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354183 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354235 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354268 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354355 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bf0314-53b2-4937-9106-f72080e6d0f1-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354386 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj58w\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-kube-api-access-kj58w\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354409 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354435 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354492 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354534 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354712 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.354910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354814 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.355054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.354969 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.355101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.355040 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.355263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.355243 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bf0314-53b2-4937-9106-f72080e6d0f1-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.356951 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.356927 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.357164 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.357145 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.398407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.393761 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.403344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.402053 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj58w\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-kube-api-access-kj58w\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.495403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.494605 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:03.851737 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.851707 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c"] Oct 31 21:31:03.851918 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:03.851829 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94bf0314_53b2_4937_9106_f72080e6d0f1.slice/crio-1917d05ce64a107f5a750157d100476acc48cd33c56b0845847b8548527e8b06 WatchSource:0}: Error finding container 1917d05ce64a107f5a750157d100476acc48cd33c56b0845847b8548527e8b06: Status 404 returned error can't find the container with id 1917d05ce64a107f5a750157d100476acc48cd33c56b0845847b8548527e8b06 Oct 31 21:31:03.855353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.855303 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:03.855533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.855510 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:03.855627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:03.855564 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:04.144789 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.144753 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:04.205373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.202756 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17258272-6b47-43d4-929b-62e74e9b3ce4-kube-api-access-whr6g\" (UniqueName: \"kubernetes.io/projected/17258272-6b47-43d4-929b-62e74e9b3ce4-kube-api-access-whr6g\") pod \"17258272-6b47-43d4-929b-62e74e9b3ce4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " Oct 31 21:31:04.205373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.202856 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-openshift-ca\") pod \"17258272-6b47-43d4-929b-62e74e9b3ce4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " Oct 31 21:31:04.205373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.202890 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-xks-ca\") pod \"17258272-6b47-43d4-929b-62e74e9b3ce4\" (UID: \"17258272-6b47-43d4-929b-62e74e9b3ce4\") " Oct 31 21:31:04.226158 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.226121 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-openshift-ca" pod "17258272-6b47-43d4-929b-62e74e9b3ce4" (UID: "17258272-6b47-43d4-929b-62e74e9b3ce4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:04.233361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.231827 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 21:31:04.233361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.231896 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 21:31:04.233361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.232242 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-xks-ca" pod "17258272-6b47-43d4-929b-62e74e9b3ce4" (UID: "17258272-6b47-43d4-929b-62e74e9b3ce4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:04.234778 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.234617 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17258272-6b47-43d4-929b-62e74e9b3ce4-kube-api-access-whr6g" pod "17258272-6b47-43d4-929b-62e74e9b3ce4" (UID: "17258272-6b47-43d4-929b-62e74e9b3ce4"). InnerVolumeSpecName "kube-api-access-whr6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:04.309111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.309058 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whr6g\" (UniqueName: \"kubernetes.io/projected/17258272-6b47-43d4-929b-62e74e9b3ce4-kube-api-access-whr6g\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:04.309111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.309106 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:04.309429 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.309123 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17258272-6b47-43d4-929b-62e74e9b3ce4-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:04.847287 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.847175 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" Oct 31 21:31:04.847491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.847175 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4" event={"ID":"17258272-6b47-43d4-929b-62e74e9b3ce4","Type":"ContainerDied","Data":"871c992294eb9bb75ee595d4efa81d5356a8f0b7ec7f4e75c7db6c8f1dcf593b"} Oct 31 21:31:04.847491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.847415 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="871c992294eb9bb75ee595d4efa81d5356a8f0b7ec7f4e75c7db6c8f1dcf593b" Oct 31 21:31:04.849540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.849475 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" event={"ID":"94bf0314-53b2-4937-9106-f72080e6d0f1","Type":"ContainerStarted","Data":"9bd82551b7551f90adbb6e2a89f4bf5c7d87e4a8896ff02b536a05a333451832"} Oct 31 21:31:04.850498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.849564 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" event={"ID":"94bf0314-53b2-4937-9106-f72080e6d0f1","Type":"ContainerStarted","Data":"1917d05ce64a107f5a750157d100476acc48cd33c56b0845847b8548527e8b06"} Oct 31 21:31:04.915203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:04.915156 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podStartSLOduration=1.915141022 podStartE2EDuration="1.915141022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:04.910314204 +0000 UTC m=+911.603271598" watchObservedRunningTime="2026-10-31 21:31:04.915141022 +0000 UTC m=+911.608098415" Oct 31 21:31:05.231879 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:05.231836 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 21:31:05.232187 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:05.232155 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 21:31:05.502407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:05.501637 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:05.502407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:05.501783 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 21:31:05.502407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:05.501836 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 21:31:06.229636 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:06.229598 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 21:31:06.229894 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:06.229661 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 21:31:06.500408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:06.497492 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 21:31:06.500408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:06.497589 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 21:31:07.599363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:07.598428 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 21:31:07.599363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:07.598501 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 21:31:07.939490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:07.923081 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2"] Oct 31 21:31:07.939490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:07.930537 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:31:07.939490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:07.935453 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:31:07.987386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:07.980260 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:31:08.524514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:08.524468 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 21:31:08.524806 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:08.524774 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 21:31:08.846407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:08.845365 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:09.046347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:09.042433 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:09.314350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:09.310493 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:31:09.530448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:09.530396 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 21:31:09.530786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:09.530750 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 21:31:09.850350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:09.849485 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:09.850900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:09.850527 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:31:10.046362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:10.043777 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:10.046362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:10.044897 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:31:10.252490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:10.252434 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:10.498346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:10.497797 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 21:31:10.498346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:10.497867 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 21:31:10.853489 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:10.853429 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:11.056596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:11.055484 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:11.083084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:11.083054 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:31:11.300514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:11.300460 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:11.325735 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:11.325690 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:11.326721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:11.326694 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:31:11.570824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:11.570716 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 21:31:11.570824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:11.570800 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 21:31:11.625522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:11.625486 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:31:11.652406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:11.652371 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:31:12.498953 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.498903 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 21:31:12.499621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.499557 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 21:31:12.760289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.760195 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc"] Oct 31 21:31:12.760812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.760796 5188 cpu_manager.go:414] "containerMap: removing container" podUID="17258272-6b47-43d4-929b-62e74e9b3ce4" containerName="revoke-keys" Oct 31 21:31:12.760859 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.760813 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17258272-6b47-43d4-929b-62e74e9b3ce4" containerName="revoke-keys" Oct 31 21:31:12.760950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.760940 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="17258272-6b47-43d4-929b-62e74e9b3ce4" containerName="revoke-keys" Oct 31 21:31:12.764721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.764687 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:12.861850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.861802 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc"] Oct 31 21:31:12.932174 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932130 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:12.932174 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932190 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:12.932534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932282 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:12.932534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932247 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" podUID="647513a8-89c8-465a-a7ca-825eefc87d63" containerName="payload-processing" containerID="cri-o://7f0516881241fdcac84be5e1a1a554ce943d2fc0c950ddd6995beb34006fce26" gracePeriod=25 Oct 31 21:31:12.932534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932415 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2d8g\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-kube-api-access-c2d8g\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:12.932534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932513 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:12.932769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932595 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:12.932769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932638 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:12.933265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932775 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:12.933265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:12.932879 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.034471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.034371 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.034471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.034460 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.034709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.034510 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.034709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.034563 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2d8g\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-kube-api-access-c2d8g\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.034709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.034642 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.034997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.034972 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.035077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.035058 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.035132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.035096 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.035132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.035121 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.035270 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.035227 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.035343 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.035303 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.035343 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.035315 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.035483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.035412 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.036152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.036123 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.037250 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.037217 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.038056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.038021 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.056149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.055779 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.056149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.055851 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2d8g\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-kube-api-access-c2d8g\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.155222 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.144932 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p"] Oct 31 21:31:13.156033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.155494 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:13.163448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.163288 5188 generic.go:356] "Generic (PLEG): container finished" podID="647513a8-89c8-465a-a7ca-825eefc87d63" containerID="7f0516881241fdcac84be5e1a1a554ce943d2fc0c950ddd6995beb34006fce26" exitCode=0 Oct 31 21:31:13.163525 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.163501 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" event={"ID":"647513a8-89c8-465a-a7ca-825eefc87d63","Type":"ContainerDied","Data":"7f0516881241fdcac84be5e1a1a554ce943d2fc0c950ddd6995beb34006fce26"} Oct 31 21:31:13.165281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.163996 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.176917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.176883 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:31:13.177381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.176968 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:31:13.177381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.177083 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:31:13.177381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.177145 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:31:13.178778 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.178742 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p"] Oct 31 21:31:13.228438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.226429 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:31:13.338540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.338380 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-provider-credentials\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-provider-credentials\") pod \"647513a8-89c8-465a-a7ca-825eefc87d63\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " Oct 31 21:31:13.338540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.338525 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-plugins-config-volume\") pod \"647513a8-89c8-465a-a7ca-825eefc87d63\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " Oct 31 21:31:13.338924 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.338891 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-routing-overlay-volume\") pod \"647513a8-89c8-465a-a7ca-825eefc87d63\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " Oct 31 21:31:13.339148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.339127 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-kube-api-access-cqcwz\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-kube-api-access-cqcwz\") pod \"647513a8-89c8-465a-a7ca-825eefc87d63\" (UID: \"647513a8-89c8-465a-a7ca-825eefc87d63\") " Oct 31 21:31:13.339287 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.338975 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-plugins-config-volume" pod "647513a8-89c8-465a-a7ca-825eefc87d63" (UID: "647513a8-89c8-465a-a7ca-825eefc87d63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:13.339447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.339317 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-routing-overlay-volume" pod "647513a8-89c8-465a-a7ca-825eefc87d63" (UID: "647513a8-89c8-465a-a7ca-825eefc87d63"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:13.339623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.339584 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9128dcf4-xfn9p\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.339830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.339811 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9128dcf4-xfn9p\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.339962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.339945 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr27k\" (UniqueName: \"kubernetes.io/projected/d8338233-eadd-47de-806c-8bae1b0617cf-kube-api-access-xr27k\") pod \"maas-api-revoke-sub-9128dcf4-xfn9p\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.340365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.340298 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:13.340365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.340342 5188 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/647513a8-89c8-465a-a7ca-825eefc87d63-routing-overlay-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:13.341255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.341222 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-provider-credentials" pod "647513a8-89c8-465a-a7ca-825eefc87d63" (UID: "647513a8-89c8-465a-a7ca-825eefc87d63"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:13.341817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.341796 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-kube-api-access-cqcwz" pod "647513a8-89c8-465a-a7ca-825eefc87d63" (UID: "647513a8-89c8-465a-a7ca-825eefc87d63"). InnerVolumeSpecName "kube-api-access-cqcwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:13.441371 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.441318 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9128dcf4-xfn9p\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.441552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.441436 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9128dcf4-xfn9p\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.441552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.441490 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr27k\" (UniqueName: \"kubernetes.io/projected/d8338233-eadd-47de-806c-8bae1b0617cf-kube-api-access-xr27k\") pod \"maas-api-revoke-sub-9128dcf4-xfn9p\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.441696 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.441655 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqcwz\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-kube-api-access-cqcwz\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:13.441696 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.441674 5188 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/647513a8-89c8-465a-a7ca-825eefc87d63-provider-credentials\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:13.442080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.442054 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9128dcf4-xfn9p\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.444397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.444365 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9128dcf4-xfn9p\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.464049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.463583 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr27k\" (UniqueName: \"kubernetes.io/projected/d8338233-eadd-47de-806c-8bae1b0617cf-kube-api-access-xr27k\") pod \"maas-api-revoke-sub-9128dcf4-xfn9p\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.486290 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.486248 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:13.498446 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.498409 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 31 21:31:13.498799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.498765 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 31 21:31:13.527531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.527490 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc"] Oct 31 21:31:13.527531 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:13.527508 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf81bcb4_599c_4aca_87ab_64f964cf6e54.slice/crio-4eb13a17757036c61446a7e1593008c6d890f235e09bd8caafdf94563137c48f WatchSource:0}: Error finding container 4eb13a17757036c61446a7e1593008c6d890f235e09bd8caafdf94563137c48f: Status 404 returned error can't find the container with id 4eb13a17757036c61446a7e1593008c6d890f235e09bd8caafdf94563137c48f Oct 31 21:31:13.530404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.530288 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:13.530516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.530422 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:13.530516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.530461 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:13.582845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.582805 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7"] Oct 31 21:31:13.583663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.583640 5188 cpu_manager.go:414] "containerMap: removing container" podUID="647513a8-89c8-465a-a7ca-825eefc87d63" containerName="payload-processing" Oct 31 21:31:13.583825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.583812 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647513a8-89c8-465a-a7ca-825eefc87d63" containerName="payload-processing" Oct 31 21:31:13.584218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.584204 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="647513a8-89c8-465a-a7ca-825eefc87d63" containerName="payload-processing" Oct 31 21:31:13.596399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.595447 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.601609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.601573 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7"] Oct 31 21:31:13.746170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.746032 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a3de07-w5gd7\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.746747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.746543 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a3de07-w5gd7\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.746747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.746622 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84pr6\" (UniqueName: \"kubernetes.io/projected/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-kube-api-access-84pr6\") pod \"maas-api-revoke-sub-34a3de07-w5gd7\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.813403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.813362 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p"] Oct 31 21:31:13.817288 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:13.816799 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8338233_eadd_47de_806c_8bae1b0617cf.slice/crio-0e2d1f9733acfefeba3b6a98e9eb07258aaf158d2586ffa7eb87dc9d6dc7040c WatchSource:0}: Error finding container 0e2d1f9733acfefeba3b6a98e9eb07258aaf158d2586ffa7eb87dc9d6dc7040c: Status 404 returned error can't find the container with id 0e2d1f9733acfefeba3b6a98e9eb07258aaf158d2586ffa7eb87dc9d6dc7040c Oct 31 21:31:13.848353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.847482 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84pr6\" (UniqueName: \"kubernetes.io/projected/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-kube-api-access-84pr6\") pod \"maas-api-revoke-sub-34a3de07-w5gd7\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.848353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.847656 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a3de07-w5gd7\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.848353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.847773 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a3de07-w5gd7\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.848774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.848652 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34a3de07-w5gd7\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.851097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.851072 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34a3de07-w5gd7\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.869217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.869186 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84pr6\" (UniqueName: \"kubernetes.io/projected/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-kube-api-access-84pr6\") pod \"maas-api-revoke-sub-34a3de07-w5gd7\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:13.938842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:13.938797 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:14.195195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.195155 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" event={"ID":"d8338233-eadd-47de-806c-8bae1b0617cf","Type":"ContainerStarted","Data":"0e2d1f9733acfefeba3b6a98e9eb07258aaf158d2586ffa7eb87dc9d6dc7040c"} Oct 31 21:31:14.198551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.198499 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" event={"ID":"647513a8-89c8-465a-a7ca-825eefc87d63","Type":"ContainerDied","Data":"52b643017a142b3b33aaa8035f996aa915058d8deaf5bcaa1a3ecbce54f49884"} Oct 31 21:31:14.198839 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.198812 5188 scope.go:122] "RemoveContainer" containerID="7f0516881241fdcac84be5e1a1a554ce943d2fc0c950ddd6995beb34006fce26" Oct 31 21:31:14.199182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.199155 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2" Oct 31 21:31:14.209902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.208963 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" event={"ID":"bf81bcb4-599c-4aca-87ab-64f964cf6e54","Type":"ContainerStarted","Data":"2ff6847a9ebe2a567252267affd167e40c7f7b4376392f88cfa1e0d585beb0ca"} Oct 31 21:31:14.209902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.209035 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" event={"ID":"bf81bcb4-599c-4aca-87ab-64f964cf6e54","Type":"ContainerStarted","Data":"4eb13a17757036c61446a7e1593008c6d890f235e09bd8caafdf94563137c48f"} Oct 31 21:31:14.255172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.255120 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2"] Oct 31 21:31:14.259123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.259077 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb987f6c7-gdbp2"] Oct 31 21:31:14.290484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.290413 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" podStartSLOduration=2.290393275 podStartE2EDuration="2.290393275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:14.289871627 +0000 UTC m=+920.982829021" watchObservedRunningTime="2026-10-31 21:31:14.290393275 +0000 UTC m=+920.983350668" Oct 31 21:31:14.327009 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:14.326967 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7"] Oct 31 21:31:15.156821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.156779 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:15.158475 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.158444 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 21:31:15.158608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.158504 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 21:31:15.255423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.255283 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" event={"ID":"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5","Type":"ContainerStarted","Data":"de6831a34757c92adba71bbe42b6b6975aef628cd352f7dace9a0299204da069"} Oct 31 21:31:15.255423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.255404 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" event={"ID":"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5","Type":"ContainerStarted","Data":"1689d15c4261c736ff3507c16e36eeb6c00148396ffdc6dc86f480a784ab7aa5"} Oct 31 21:31:15.257201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.257171 5188 generic.go:356] "Generic (PLEG): container finished" podID="d8338233-eadd-47de-806c-8bae1b0617cf" containerID="c09bd1371af7437c2cd0797c7a6e6940afae73fc618f3e6c0ada884beac6b50e" exitCode=0 Oct 31 21:31:15.257361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.257252 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" event={"ID":"d8338233-eadd-47de-806c-8bae1b0617cf","Type":"ContainerDied","Data":"c09bd1371af7437c2cd0797c7a6e6940afae73fc618f3e6c0ada884beac6b50e"} Oct 31 21:31:15.361599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.361531 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" podStartSLOduration=2.3615140439999998 podStartE2EDuration="2.361514044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:15.322024175 +0000 UTC m=+922.014981570" watchObservedRunningTime="2026-10-31 21:31:15.361514044 +0000 UTC m=+922.054471491" Oct 31 21:31:15.491659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.491610 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647513a8-89c8-465a-a7ca-825eefc87d63" path="/var/lib/kubelet/pods/647513a8-89c8-465a-a7ca-825eefc87d63/volumes" Oct 31 21:31:15.538637 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.538514 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:31:15.538931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.538893 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:15.928474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.928440 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:15.929198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.929166 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:15.930268 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:15.930246 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:31:16.156605 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.156571 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 21:31:16.156791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.156631 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 21:31:16.269366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.269233 5188 generic.go:356] "Generic (PLEG): container finished" podID="d360b3a3-537c-4fa1-90c7-72ad0d76d6f5" containerID="de6831a34757c92adba71bbe42b6b6975aef628cd352f7dace9a0299204da069" exitCode=0 Oct 31 21:31:16.269807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.269386 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" event={"ID":"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5","Type":"ContainerDied","Data":"de6831a34757c92adba71bbe42b6b6975aef628cd352f7dace9a0299204da069"} Oct 31 21:31:16.543933 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.543901 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:16.653714 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.653661 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8338233-eadd-47de-806c-8bae1b0617cf-kube-api-access-xr27k\" (UniqueName: \"kubernetes.io/projected/d8338233-eadd-47de-806c-8bae1b0617cf-kube-api-access-xr27k\") pod \"d8338233-eadd-47de-806c-8bae1b0617cf\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " Oct 31 21:31:16.653714 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.653717 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-xks-ca\") pod \"d8338233-eadd-47de-806c-8bae1b0617cf\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " Oct 31 21:31:16.653973 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.653760 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-openshift-ca\") pod \"d8338233-eadd-47de-806c-8bae1b0617cf\" (UID: \"d8338233-eadd-47de-806c-8bae1b0617cf\") " Oct 31 21:31:16.654966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.654933 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-openshift-ca" pod "d8338233-eadd-47de-806c-8bae1b0617cf" (UID: "d8338233-eadd-47de-806c-8bae1b0617cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:16.713397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.709875 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8338233-eadd-47de-806c-8bae1b0617cf-kube-api-access-xr27k" pod "d8338233-eadd-47de-806c-8bae1b0617cf" (UID: "d8338233-eadd-47de-806c-8bae1b0617cf"). InnerVolumeSpecName "kube-api-access-xr27k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:16.724084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.724016 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-xks-ca" pod "d8338233-eadd-47de-806c-8bae1b0617cf" (UID: "d8338233-eadd-47de-806c-8bae1b0617cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:16.755667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.755627 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr27k\" (UniqueName: \"kubernetes.io/projected/d8338233-eadd-47de-806c-8bae1b0617cf-kube-api-access-xr27k\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:16.755667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.755663 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:16.755667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:16.755673 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8338233-eadd-47de-806c-8bae1b0617cf-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:17.156241 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.156202 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 21:31:17.156482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.156260 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 21:31:17.314350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.313390 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" Oct 31 21:31:17.321391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.318359 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p" event={"ID":"d8338233-eadd-47de-806c-8bae1b0617cf","Type":"ContainerDied","Data":"0e2d1f9733acfefeba3b6a98e9eb07258aaf158d2586ffa7eb87dc9d6dc7040c"} Oct 31 21:31:17.321391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.318443 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e2d1f9733acfefeba3b6a98e9eb07258aaf158d2586ffa7eb87dc9d6dc7040c" Oct 31 21:31:17.647637 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.647604 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:17.678433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.672849 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-xks-ca\") pod \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " Oct 31 21:31:17.678433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.673094 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-kube-api-access-84pr6\" (UniqueName: \"kubernetes.io/projected/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-kube-api-access-84pr6\") pod \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " Oct 31 21:31:17.678433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.673178 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-openshift-ca\") pod \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\" (UID: \"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5\") " Oct 31 21:31:17.678433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.674294 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-openshift-ca" pod "d360b3a3-537c-4fa1-90c7-72ad0d76d6f5" (UID: "d360b3a3-537c-4fa1-90c7-72ad0d76d6f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:17.678433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.677402 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-xks-ca" pod "d360b3a3-537c-4fa1-90c7-72ad0d76d6f5" (UID: "d360b3a3-537c-4fa1-90c7-72ad0d76d6f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:17.705101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.705062 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-kube-api-access-84pr6" pod "d360b3a3-537c-4fa1-90c7-72ad0d76d6f5" (UID: "d360b3a3-537c-4fa1-90c7-72ad0d76d6f5"). InnerVolumeSpecName "kube-api-access-84pr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:17.780365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.776356 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:17.780365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.776419 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84pr6\" (UniqueName: \"kubernetes.io/projected/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-kube-api-access-84pr6\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:17.780365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:17.776442 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:18.155942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:18.155902 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 21:31:18.156132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:18.155967 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 21:31:18.321362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:18.321308 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" event={"ID":"d360b3a3-537c-4fa1-90c7-72ad0d76d6f5","Type":"ContainerDied","Data":"1689d15c4261c736ff3507c16e36eeb6c00148396ffdc6dc86f480a784ab7aa5"} Oct 31 21:31:18.321362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:18.321354 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7" Oct 31 21:31:18.321833 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:18.321387 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1689d15c4261c736ff3507c16e36eeb6c00148396ffdc6dc86f480a784ab7aa5" Oct 31 21:31:19.156790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:19.156747 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 21:31:19.157004 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:19.156810 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 21:31:20.157033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:20.156994 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 21:31:20.157487 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:20.157056 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 21:31:21.161381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:21.161339 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:21.161937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:21.161911 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:21.162896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:21.162874 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:31:23.538937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.538897 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2"] Oct 31 21:31:23.539502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.539485 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d360b3a3-537c-4fa1-90c7-72ad0d76d6f5" containerName="revoke-keys" Oct 31 21:31:23.539566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.539503 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d360b3a3-537c-4fa1-90c7-72ad0d76d6f5" containerName="revoke-keys" Oct 31 21:31:23.539566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.539518 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d8338233-eadd-47de-806c-8bae1b0617cf" containerName="revoke-keys" Oct 31 21:31:23.539566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.539523 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8338233-eadd-47de-806c-8bae1b0617cf" containerName="revoke-keys" Oct 31 21:31:23.539665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.539640 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d360b3a3-537c-4fa1-90c7-72ad0d76d6f5" containerName="revoke-keys" Oct 31 21:31:23.539665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.539649 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8338233-eadd-47de-806c-8bae1b0617cf" containerName="revoke-keys" Oct 31 21:31:23.543015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.542994 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.567440 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.567411 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:31:23.567639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.567433 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:31:23.567639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.567446 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:31:23.567851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.567433 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:31:23.569736 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.569708 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2"] Oct 31 21:31:23.665895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.665857 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add1ab35-lgxs2\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.666064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.666024 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add1ab35-lgxs2\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.666209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.666191 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgxsg\" (UniqueName: \"kubernetes.io/projected/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-kube-api-access-vgxsg\") pod \"maas-api-revoke-sub-add1ab35-lgxs2\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.767257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.767209 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgxsg\" (UniqueName: \"kubernetes.io/projected/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-kube-api-access-vgxsg\") pod \"maas-api-revoke-sub-add1ab35-lgxs2\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.767559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.767526 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add1ab35-lgxs2\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.767705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.767673 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add1ab35-lgxs2\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.768537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.768508 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add1ab35-lgxs2\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.770661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.770632 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add1ab35-lgxs2\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.796582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.796500 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgxsg\" (UniqueName: \"kubernetes.io/projected/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-kube-api-access-vgxsg\") pod \"maas-api-revoke-sub-add1ab35-lgxs2\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:23.858096 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:23.858053 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:24.171628 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:24.171582 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5adc2f7_42fc_4615_9732_cc7b9c9d8dcd.slice/crio-49e4bde92dd75a71dd903e3a2f59d465584e983f92d9a55605fe69868839327c WatchSource:0}: Error finding container 49e4bde92dd75a71dd903e3a2f59d465584e983f92d9a55605fe69868839327c: Status 404 returned error can't find the container with id 49e4bde92dd75a71dd903e3a2f59d465584e983f92d9a55605fe69868839327c Oct 31 21:31:24.171628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:24.171609 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2"] Oct 31 21:31:24.370823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:24.370779 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" event={"ID":"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd","Type":"ContainerStarted","Data":"49e4bde92dd75a71dd903e3a2f59d465584e983f92d9a55605fe69868839327c"} Oct 31 21:31:25.422788 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:25.422752 5188 generic.go:356] "Generic (PLEG): container finished" podID="f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd" containerID="a5709e82c3419c453403f480f7be64a214bfc7744b29d2504e0d0240de2d2e53" exitCode=0 Oct 31 21:31:25.423454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:25.423425 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" event={"ID":"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd","Type":"ContainerDied","Data":"a5709e82c3419c453403f480f7be64a214bfc7744b29d2504e0d0240de2d2e53"} Oct 31 21:31:27.104742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:27.104684 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:27.302559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:27.302491 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:27.695469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:27.695410 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:31:28.063111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.060745 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:28.067150 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.067114 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:28.146350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.133707 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:28.323700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.323594 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:28.384399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.384363 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-openshift-ca\") pod \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " Oct 31 21:31:28.402136 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.402106 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-kube-api-access-vgxsg\" (UniqueName: \"kubernetes.io/projected/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-kube-api-access-vgxsg\") pod \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " Oct 31 21:31:28.402491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.402467 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-xks-ca\") pod \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\" (UID: \"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd\") " Oct 31 21:31:28.404017 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.401903 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-openshift-ca" pod "f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd" (UID: "f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:28.404949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.404928 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:28.433362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.433289 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:31:28.435523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.435500 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:31:28.616541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.616492 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-xks-ca" pod "f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd" (UID: "f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:28.776833 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.760573 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:28.860501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.860451 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:31:28.881740 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:28.861474 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:31:29.006349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.003778 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-kube-api-access-vgxsg" pod "f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd" (UID: "f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd"). InnerVolumeSpecName "kube-api-access-vgxsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:29.006349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.006240 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgxsg\" (UniqueName: \"kubernetes.io/projected/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd-kube-api-access-vgxsg\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:29.075353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.073444 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 21:31:29.075353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.074122 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:31:29.206406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.196446 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:29.329609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.326412 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:29.542417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.538400 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" Oct 31 21:31:29.751349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.745613 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2" event={"ID":"f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd","Type":"ContainerDied","Data":"49e4bde92dd75a71dd903e3a2f59d465584e983f92d9a55605fe69868839327c"} Oct 31 21:31:29.751349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.745686 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e4bde92dd75a71dd903e3a2f59d465584e983f92d9a55605fe69868839327c" Oct 31 21:31:29.862364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.861999 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:31:30.003435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.999455 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cts4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:31:30.003435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:29.999534 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:30.003717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.003455 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:31:30.003717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.003528 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:30.032424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.030079 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:31:30.096397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.096361 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4"] Oct 31 21:31:30.097503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.097475 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd" containerName="revoke-keys" Oct 31 21:31:30.097648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.097634 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd" containerName="revoke-keys" Oct 31 21:31:30.097931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.097913 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd" containerName="revoke-keys" Oct 31 21:31:30.133734 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.133655 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.148717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.147784 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:31:30.148717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.148547 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:31:30.148717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.148609 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:31:30.148717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.148668 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:31:30.185408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.162739 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cee038-t64b4\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.185408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.162840 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slx6\" (UniqueName: \"kubernetes.io/projected/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-kube-api-access-6slx6\") pod \"maas-api-revoke-sub-d9cee038-t64b4\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.185408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.162879 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cee038-t64b4\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.225030 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.222147 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:30.275347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.267062 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slx6\" (UniqueName: \"kubernetes.io/projected/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-kube-api-access-6slx6\") pod \"maas-api-revoke-sub-d9cee038-t64b4\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.331918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.275695 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cee038-t64b4\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.332295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.331303 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:31:30.357562 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.357522 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cee038-t64b4\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.358671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.358629 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9cee038-t64b4\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.360303 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.360278 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4"] Oct 31 21:31:30.490133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.490009 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9cee038-t64b4\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.629250 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.629176 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slx6\" (UniqueName: \"kubernetes.io/projected/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-kube-api-access-6slx6\") pod \"maas-api-revoke-sub-d9cee038-t64b4\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.834544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.834432 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 31 21:31:30.839560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.839519 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:30.883471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:30.883421 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:31.029495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:31.029440 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:31.234652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:31.234560 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 31 21:31:31.339696 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:31.339639 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:31:31.784288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:31.784227 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-f9pgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:31:31.784613 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:31.784580 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:32.061236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:32.061137 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:32.062205 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:32.062066 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:31:32.070465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:32.070375 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:31:32.070899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:32.070874 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:31:32.079894 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:32.079859 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:31:32.101377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:32.100430 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:31:32.119939 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:32.119896 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:31:32.187978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:32.187931 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4"] Oct 31 21:31:32.540460 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:31:32.540417 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:31:33.092960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:33.092926 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" event={"ID":"6d0ec435-3c2a-442f-b2f5-93994c9d5e73","Type":"ContainerStarted","Data":"5f7886601fe680b4b8a4c243020f457fb115018ac93d7d811bb37794242c1f64"} Oct 31 21:31:33.093176 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:33.092982 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" event={"ID":"6d0ec435-3c2a-442f-b2f5-93994c9d5e73","Type":"ContainerStarted","Data":"61bc559f0a81eb694cae01421b38c40d32300b8d1ef81df96698f895b575f7ae"} Oct 31 21:31:33.131807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:33.131742 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" podStartSLOduration=3.131726706 podStartE2EDuration="3.131726706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:33.128238133 +0000 UTC m=+939.821195526" watchObservedRunningTime="2026-10-31 21:31:33.131726706 +0000 UTC m=+939.824684099" Oct 31 21:31:34.104241 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:34.104154 5188 generic.go:356] "Generic (PLEG): container finished" podID="6d0ec435-3c2a-442f-b2f5-93994c9d5e73" containerID="5f7886601fe680b4b8a4c243020f457fb115018ac93d7d811bb37794242c1f64" exitCode=0 Oct 31 21:31:34.104673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:34.104239 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" event={"ID":"6d0ec435-3c2a-442f-b2f5-93994c9d5e73","Type":"ContainerDied","Data":"5f7886601fe680b4b8a4c243020f457fb115018ac93d7d811bb37794242c1f64"} Oct 31 21:31:35.304016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.303986 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:35.367394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.367260 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-kube-api-access-6slx6\" (UniqueName: \"kubernetes.io/projected/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-kube-api-access-6slx6\") pod \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " Oct 31 21:31:35.367606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.367560 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-openshift-ca\") pod \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " Oct 31 21:31:35.367698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.367672 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-xks-ca\") pod \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\" (UID: \"6d0ec435-3c2a-442f-b2f5-93994c9d5e73\") " Oct 31 21:31:35.368123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.367976 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-openshift-ca" pod "6d0ec435-3c2a-442f-b2f5-93994c9d5e73" (UID: "6d0ec435-3c2a-442f-b2f5-93994c9d5e73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:35.368288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.368259 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:35.369722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.369693 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-kube-api-access-6slx6" pod "6d0ec435-3c2a-442f-b2f5-93994c9d5e73" (UID: "6d0ec435-3c2a-442f-b2f5-93994c9d5e73"). InnerVolumeSpecName "kube-api-access-6slx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:35.369920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.369901 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-xks-ca" pod "6d0ec435-3c2a-442f-b2f5-93994c9d5e73" (UID: "6d0ec435-3c2a-442f-b2f5-93994c9d5e73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:35.469358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.469304 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:35.469358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:35.469353 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6slx6\" (UniqueName: \"kubernetes.io/projected/6d0ec435-3c2a-442f-b2f5-93994c9d5e73-kube-api-access-6slx6\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:36.132012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.131968 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" event={"ID":"6d0ec435-3c2a-442f-b2f5-93994c9d5e73","Type":"ContainerDied","Data":"61bc559f0a81eb694cae01421b38c40d32300b8d1ef81df96698f895b575f7ae"} Oct 31 21:31:36.132012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.132005 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4" Oct 31 21:31:36.132289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.132047 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61bc559f0a81eb694cae01421b38c40d32300b8d1ef81df96698f895b575f7ae" Oct 31 21:31:36.371973 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.371933 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv"] Oct 31 21:31:36.372708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.372683 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6d0ec435-3c2a-442f-b2f5-93994c9d5e73" containerName="revoke-keys" Oct 31 21:31:36.372840 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.372704 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0ec435-3c2a-442f-b2f5-93994c9d5e73" containerName="revoke-keys" Oct 31 21:31:36.372913 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.372897 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0ec435-3c2a-442f-b2f5-93994c9d5e73" containerName="revoke-keys" Oct 31 21:31:36.376852 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.376827 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.391635 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.391597 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:31:36.391830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.391678 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:31:36.391830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.391715 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:31:36.391830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.391810 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:31:36.397514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.397479 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv"] Oct 31 21:31:36.484133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.484092 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f91a1c-lxnnv\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.484360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.484149 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f91a1c-lxnnv\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.484436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.484352 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhpfd\" (UniqueName: \"kubernetes.io/projected/e86b8fab-cad6-4283-956f-c17e93c928c3-kube-api-access-vhpfd\") pod \"maas-api-revoke-sub-35f91a1c-lxnnv\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.586177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.586139 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhpfd\" (UniqueName: \"kubernetes.io/projected/e86b8fab-cad6-4283-956f-c17e93c928c3-kube-api-access-vhpfd\") pod \"maas-api-revoke-sub-35f91a1c-lxnnv\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.586416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.586294 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f91a1c-lxnnv\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.586416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.586339 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f91a1c-lxnnv\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.587147 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.587119 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f91a1c-lxnnv\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.589281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.589253 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f91a1c-lxnnv\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.607885 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.607849 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhpfd\" (UniqueName: \"kubernetes.io/projected/e86b8fab-cad6-4283-956f-c17e93c928c3-kube-api-access-vhpfd\") pod \"maas-api-revoke-sub-35f91a1c-lxnnv\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:36.693209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:36.693105 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:37.010762 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:37.010729 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86b8fab_cad6_4283_956f_c17e93c928c3.slice/crio-de998be14d26f99b70d3b8c7d30bace63b7d8eb97c37324c07ce5af5ebe6b869 WatchSource:0}: Error finding container de998be14d26f99b70d3b8c7d30bace63b7d8eb97c37324c07ce5af5ebe6b869: Status 404 returned error can't find the container with id de998be14d26f99b70d3b8c7d30bace63b7d8eb97c37324c07ce5af5ebe6b869 Oct 31 21:31:37.011877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:37.011852 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv"] Oct 31 21:31:37.142394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:37.142350 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" event={"ID":"e86b8fab-cad6-4283-956f-c17e93c928c3","Type":"ContainerStarted","Data":"de998be14d26f99b70d3b8c7d30bace63b7d8eb97c37324c07ce5af5ebe6b869"} Oct 31 21:31:38.151849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:38.151750 5188 generic.go:356] "Generic (PLEG): container finished" podID="e86b8fab-cad6-4283-956f-c17e93c928c3" containerID="b1fdb5df24b735ddfb6a68f100665caeea6d0afb8aa4a1f62b78f55a18063afc" exitCode=0 Oct 31 21:31:38.151849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:38.151818 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" event={"ID":"e86b8fab-cad6-4283-956f-c17e93c928c3","Type":"ContainerDied","Data":"b1fdb5df24b735ddfb6a68f100665caeea6d0afb8aa4a1f62b78f55a18063afc"} Oct 31 21:31:39.369817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.369777 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:39.429541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.429441 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86b8fab-cad6-4283-956f-c17e93c928c3-kube-api-access-vhpfd\" (UniqueName: \"kubernetes.io/projected/e86b8fab-cad6-4283-956f-c17e93c928c3-kube-api-access-vhpfd\") pod \"e86b8fab-cad6-4283-956f-c17e93c928c3\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " Oct 31 21:31:39.429541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.429522 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-xks-ca\") pod \"e86b8fab-cad6-4283-956f-c17e93c928c3\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " Oct 31 21:31:39.429799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.429748 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-openshift-ca\") pod \"e86b8fab-cad6-4283-956f-c17e93c928c3\" (UID: \"e86b8fab-cad6-4283-956f-c17e93c928c3\") " Oct 31 21:31:39.430577 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.430545 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-openshift-ca" pod "e86b8fab-cad6-4283-956f-c17e93c928c3" (UID: "e86b8fab-cad6-4283-956f-c17e93c928c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:39.432524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.432493 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-xks-ca" pod "e86b8fab-cad6-4283-956f-c17e93c928c3" (UID: "e86b8fab-cad6-4283-956f-c17e93c928c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:39.432643 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.432560 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86b8fab-cad6-4283-956f-c17e93c928c3-kube-api-access-vhpfd" pod "e86b8fab-cad6-4283-956f-c17e93c928c3" (UID: "e86b8fab-cad6-4283-956f-c17e93c928c3"). InnerVolumeSpecName "kube-api-access-vhpfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:39.531424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.531388 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:39.531424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.531419 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhpfd\" (UniqueName: \"kubernetes.io/projected/e86b8fab-cad6-4283-956f-c17e93c928c3-kube-api-access-vhpfd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:39.531424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:39.531429 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b8fab-cad6-4283-956f-c17e93c928c3-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:40.168919 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.168887 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" Oct 31 21:31:40.169119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.168921 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv" event={"ID":"e86b8fab-cad6-4283-956f-c17e93c928c3","Type":"ContainerDied","Data":"de998be14d26f99b70d3b8c7d30bace63b7d8eb97c37324c07ce5af5ebe6b869"} Oct 31 21:31:40.169119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.168980 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de998be14d26f99b70d3b8c7d30bace63b7d8eb97c37324c07ce5af5ebe6b869" Oct 31 21:31:40.485029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.484946 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk"] Oct 31 21:31:40.485602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.485587 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e86b8fab-cad6-4283-956f-c17e93c928c3" containerName="revoke-keys" Oct 31 21:31:40.485658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.485603 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86b8fab-cad6-4283-956f-c17e93c928c3" containerName="revoke-keys" Oct 31 21:31:40.485786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.485774 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86b8fab-cad6-4283-956f-c17e93c928c3" containerName="revoke-keys" Oct 31 21:31:40.489355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.489335 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.504217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.504146 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:31:40.511339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.509988 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk"] Oct 31 21:31:40.514080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.513989 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:31:40.514080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.514078 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:31:40.514349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.514169 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:31:40.543683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.543633 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92704f8a-kkkvk\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.543891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.543696 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92704f8a-kkkvk\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.543971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.543887 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76bt\" (UniqueName: \"kubernetes.io/projected/bad0f03a-910a-4772-9c6a-9c7624c81da4-kube-api-access-s76bt\") pod \"maas-api-revoke-sub-92704f8a-kkkvk\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.645081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.645043 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92704f8a-kkkvk\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.645255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.645099 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92704f8a-kkkvk\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.645255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.645216 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76bt\" (UniqueName: \"kubernetes.io/projected/bad0f03a-910a-4772-9c6a-9c7624c81da4-kube-api-access-s76bt\") pod \"maas-api-revoke-sub-92704f8a-kkkvk\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.645933 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.645904 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92704f8a-kkkvk\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.647669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.647637 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92704f8a-kkkvk\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.667387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.667355 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76bt\" (UniqueName: \"kubernetes.io/projected/bad0f03a-910a-4772-9c6a-9c7624c81da4-kube-api-access-s76bt\") pod \"maas-api-revoke-sub-92704f8a-kkkvk\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:40.806318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:40.806216 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:41.118004 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:41.117971 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad0f03a_910a_4772_9c6a_9c7624c81da4.slice/crio-28173c94f2f7d04dbae586b00524b941c22bd8a97a51dbe49ad89bf130b2ec0c WatchSource:0}: Error finding container 28173c94f2f7d04dbae586b00524b941c22bd8a97a51dbe49ad89bf130b2ec0c: Status 404 returned error can't find the container with id 28173c94f2f7d04dbae586b00524b941c22bd8a97a51dbe49ad89bf130b2ec0c Oct 31 21:31:41.118191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:41.118165 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk"] Oct 31 21:31:41.178247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:41.178209 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" event={"ID":"bad0f03a-910a-4772-9c6a-9c7624c81da4","Type":"ContainerStarted","Data":"28173c94f2f7d04dbae586b00524b941c22bd8a97a51dbe49ad89bf130b2ec0c"} Oct 31 21:31:42.190853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:42.190815 5188 generic.go:356] "Generic (PLEG): container finished" podID="bad0f03a-910a-4772-9c6a-9c7624c81da4" containerID="3f8c506609f1174e513c1d86267049c5b655c89617c23fbe4c94a625be91eff1" exitCode=0 Oct 31 21:31:42.191284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:42.190859 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" event={"ID":"bad0f03a-910a-4772-9c6a-9c7624c81da4","Type":"ContainerDied","Data":"3f8c506609f1174e513c1d86267049c5b655c89617c23fbe4c94a625be91eff1"} Oct 31 21:31:43.372308 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.372247 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:43.487244 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.487130 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad0f03a-910a-4772-9c6a-9c7624c81da4-kube-api-access-s76bt\" (UniqueName: \"kubernetes.io/projected/bad0f03a-910a-4772-9c6a-9c7624c81da4-kube-api-access-s76bt\") pod \"bad0f03a-910a-4772-9c6a-9c7624c81da4\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " Oct 31 21:31:43.487476 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.487294 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-openshift-ca\") pod \"bad0f03a-910a-4772-9c6a-9c7624c81da4\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " Oct 31 21:31:43.487476 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.487360 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-xks-ca\") pod \"bad0f03a-910a-4772-9c6a-9c7624c81da4\" (UID: \"bad0f03a-910a-4772-9c6a-9c7624c81da4\") " Oct 31 21:31:43.487686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.487652 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-openshift-ca" pod "bad0f03a-910a-4772-9c6a-9c7624c81da4" (UID: "bad0f03a-910a-4772-9c6a-9c7624c81da4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:43.487830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.487805 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:43.489573 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.489542 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-xks-ca" pod "bad0f03a-910a-4772-9c6a-9c7624c81da4" (UID: "bad0f03a-910a-4772-9c6a-9c7624c81da4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:43.489652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.489564 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad0f03a-910a-4772-9c6a-9c7624c81da4-kube-api-access-s76bt" pod "bad0f03a-910a-4772-9c6a-9c7624c81da4" (UID: "bad0f03a-910a-4772-9c6a-9c7624c81da4"). InnerVolumeSpecName "kube-api-access-s76bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:43.589491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.589450 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bad0f03a-910a-4772-9c6a-9c7624c81da4-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:43.589491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:43.589485 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s76bt\" (UniqueName: \"kubernetes.io/projected/bad0f03a-910a-4772-9c6a-9c7624c81da4-kube-api-access-s76bt\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:44.035058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.035019 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76"] Oct 31 21:31:44.043355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.036226 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bad0f03a-910a-4772-9c6a-9c7624c81da4" containerName="revoke-keys" Oct 31 21:31:44.043703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.043662 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad0f03a-910a-4772-9c6a-9c7624c81da4" containerName="revoke-keys" Oct 31 21:31:44.044144 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.044119 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad0f03a-910a-4772-9c6a-9c7624c81da4" containerName="revoke-keys" Oct 31 21:31:44.097411 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.064779 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.097411 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.072667 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76"] Oct 31 21:31:44.216582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.209745 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.216582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.209813 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.216582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.209930 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.216582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.210024 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36232b3-9979-41d1-97fe-518e3a25ac76-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.216582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.211401 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.216582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.211448 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.216582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.211605 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.216582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.211635 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kksc\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-kube-api-access-6kksc\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.216582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.211675 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.224050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.224001 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" event={"ID":"bad0f03a-910a-4772-9c6a-9c7624c81da4","Type":"ContainerDied","Data":"28173c94f2f7d04dbae586b00524b941c22bd8a97a51dbe49ad89bf130b2ec0c"} Oct 31 21:31:44.224339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.224305 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28173c94f2f7d04dbae586b00524b941c22bd8a97a51dbe49ad89bf130b2ec0c" Oct 31 21:31:44.224507 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.224493 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.313525 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36232b3-9979-41d1-97fe-518e3a25ac76-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.313641 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.313673 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.313709 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.313734 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kksc\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-kube-api-access-6kksc\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.313767 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.313813 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.313867 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.313996 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.321399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.315609 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36232b3-9979-41d1-97fe-518e3a25ac76-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.322074 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.321754 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.322120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.322078 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.322484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.322458 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.322637 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.322615 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.328372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.325245 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.328372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.325759 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.380345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.372357 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kksc\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-kube-api-access-6kksc\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.390345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.383953 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.422348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.422071 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:44.923348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.906864 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76"] Oct 31 21:31:44.942381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.938077 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:44.942381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.938181 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:44.942381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:44.938227 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:45.330665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:45.325568 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" event={"ID":"c36232b3-9979-41d1-97fe-518e3a25ac76","Type":"ContainerStarted","Data":"f8e04d7522c8495d3f908743b53a9501b7b283a123c6243d03848273916d40ff"} Oct 31 21:31:45.868501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:45.868455 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:46.407808 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:46.407753 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" event={"ID":"c36232b3-9979-41d1-97fe-518e3a25ac76","Type":"ContainerStarted","Data":"c37842e167782b64f36a50ea205bd1daa30893f2394de99e8255e8c38dc18b0b"} Oct 31 21:31:46.425894 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:46.425858 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:46.426231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:46.426203 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 21:31:46.426471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:46.426440 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 21:31:46.474249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:46.474168 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podStartSLOduration=3.474145805 podStartE2EDuration="3.474145805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:46.461616099 +0000 UTC m=+953.154573529" watchObservedRunningTime="2026-10-31 21:31:46.474145805 +0000 UTC m=+953.167103200" Oct 31 21:31:47.423036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:47.422992 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 21:31:47.423655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:47.423621 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 21:31:48.315908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.315849 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d"] Oct 31 21:31:48.323899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.323866 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.330367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.329549 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhh6k\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-kube-api-access-zhh6k\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.330367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.329676 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.330367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.329730 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.330367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.329749 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0af82a-8a48-4758-97ca-48ddb633fda8-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.330367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.329776 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.330367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.329799 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.330367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.329945 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.330367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.330036 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.330367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.330108 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.349261 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.349224 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d"] Oct 31 21:31:48.349520 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.349507 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk"] Oct 31 21:31:48.354927 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.354902 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.387454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.367607 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk"] Oct 31 21:31:48.424563 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.424526 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 21:31:48.425216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.425181 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 21:31:48.431799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.431773 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.432033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.432012 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.432353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.432246 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.432764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.432738 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.432920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.432903 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0af82a-8a48-4758-97ca-48ddb633fda8-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.433034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433020 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.433148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433132 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.433246 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433231 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.433375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433355 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.433505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433485 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.433641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433626 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.433805 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433789 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.433933 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433917 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.434080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.434064 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.434209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.434195 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e1f58a-9117-4538-abe5-c8377b9f5434-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.434293 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433406 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.434416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.433646 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.434530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.434505 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhh6k\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-kube-api-access-zhh6k\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.434598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.434585 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.434650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.434625 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2khw\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-kube-api-access-r2khw\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.434703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.434654 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.435160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.435114 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0af82a-8a48-4758-97ca-48ddb633fda8-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.435239 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.435153 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.436811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.436792 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.437163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.437137 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.457529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.457486 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.461910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.461873 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhh6k\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-kube-api-access-zhh6k\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.536440 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.536352 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.536694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.536455 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.536694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.536489 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.536904 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.536870 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.536976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.536940 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.537031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.537000 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.537093 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.537053 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e1f58a-9117-4538-abe5-c8377b9f5434-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.537162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.537114 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.537162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.536870 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.537162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.537138 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2khw\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-kube-api-access-r2khw\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.537310 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.537169 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.537310 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.537246 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.537773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.537745 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e1f58a-9117-4538-abe5-c8377b9f5434-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.538019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.537982 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.540266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.540239 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.542549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.542516 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.564631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.559353 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.564631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.559459 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2khw\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-kube-api-access-r2khw\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:48.651974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.651935 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:48.714405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:48.714364 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:49.031875 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:49.031834 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0af82a_8a48_4758_97ca_48ddb633fda8.slice/crio-7c1a51241a571f19859828ceafaee874b2fb263a94f9ef742f6a18d900c21113 WatchSource:0}: Error finding container 7c1a51241a571f19859828ceafaee874b2fb263a94f9ef742f6a18d900c21113: Status 404 returned error can't find the container with id 7c1a51241a571f19859828ceafaee874b2fb263a94f9ef742f6a18d900c21113 Oct 31 21:31:49.032060 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.032038 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d"] Oct 31 21:31:49.034483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.034452 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:49.034559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.034518 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:49.034559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.034552 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:49.112602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.112439 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk"] Oct 31 21:31:49.126182 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:49.125195 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e1f58a_9117_4538_abe5_c8377b9f5434.slice/crio-da5270d1fe699f2d1a864437c102137e017c7fc26b9a9feebb12c22980cdcb3b WatchSource:0}: Error finding container da5270d1fe699f2d1a864437c102137e017c7fc26b9a9feebb12c22980cdcb3b: Status 404 returned error can't find the container with id da5270d1fe699f2d1a864437c102137e017c7fc26b9a9feebb12c22980cdcb3b Oct 31 21:31:49.129094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.129044 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:49.129213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.129149 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:49.129213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.129202 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:49.425591 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.422888 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 21:31:49.425591 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.422955 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 21:31:49.444821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.443902 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" event={"ID":"4e0af82a-8a48-4758-97ca-48ddb633fda8","Type":"ContainerStarted","Data":"962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd"} Oct 31 21:31:49.444821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.443976 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" event={"ID":"4e0af82a-8a48-4758-97ca-48ddb633fda8","Type":"ContainerStarted","Data":"7c1a51241a571f19859828ceafaee874b2fb263a94f9ef742f6a18d900c21113"} Oct 31 21:31:49.448372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.448303 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" event={"ID":"08e1f58a-9117-4538-abe5-c8377b9f5434","Type":"ContainerStarted","Data":"a82dc61a344c5350fd08c6677630453f34b0fc3b086533ae1ddd257c75ebd116"} Oct 31 21:31:49.448516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.448397 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" event={"ID":"08e1f58a-9117-4538-abe5-c8377b9f5434","Type":"ContainerStarted","Data":"da5270d1fe699f2d1a864437c102137e017c7fc26b9a9feebb12c22980cdcb3b"} Oct 31 21:31:49.569072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.569008 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podStartSLOduration=1.5689934559999998 podStartE2EDuration="1.568993456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:49.523694425 +0000 UTC m=+956.216651818" watchObservedRunningTime="2026-10-31 21:31:49.568993456 +0000 UTC m=+956.261950848" Oct 31 21:31:49.570986 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.570932 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podStartSLOduration=1.570922396 podStartE2EDuration="1.570922396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:49.565912558 +0000 UTC m=+956.258869951" watchObservedRunningTime="2026-10-31 21:31:49.570922396 +0000 UTC m=+956.263879873" Oct 31 21:31:49.652770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.652734 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:49.715802 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:49.715704 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:50.304348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.303445 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc"] Oct 31 21:31:50.329349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.327737 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:50.478347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.399969 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:31:50.522248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.410080 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:31:50.522522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.410186 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:31:50.522813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.410247 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:31:50.522946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.444472 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 21:31:50.547392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.526525 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 21:31:50.778348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.479382 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6593169-vs8hc\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:50.778784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.778751 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf64l\" (UniqueName: \"kubernetes.io/projected/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-kube-api-access-mf64l\") pod \"maas-api-revoke-sub-f6593169-vs8hc\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:50.816622 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.816574 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6593169-vs8hc\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:50.819960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.720025 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:31:50.820275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.820240 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 21:31:50.821036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.749552 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 21:31:50.821209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.821186 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 21:31:50.821380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.570699 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc"] Oct 31 21:31:50.990407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:50.973375 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6593169-vs8hc\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:51.055508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.055467 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf64l\" (UniqueName: \"kubernetes.io/projected/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-kube-api-access-mf64l\") pod \"maas-api-revoke-sub-f6593169-vs8hc\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:51.092574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.092525 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6593169-vs8hc\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:51.120348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.029938 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6593169-vs8hc\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:51.237350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.236647 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6593169-vs8hc\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:51.237530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.237373 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf64l\" (UniqueName: \"kubernetes.io/projected/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-kube-api-access-mf64l\") pod \"maas-api-revoke-sub-f6593169-vs8hc\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:51.383350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.381792 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:51.462142 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.462020 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 21:31:51.462142 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.462075 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 21:31:51.658406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.656853 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 21:31:51.658406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.656921 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 21:31:51.715840 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.715711 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:31:51.715840 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:51.715777 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 21:31:52.251350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:52.250344 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc"] Oct 31 21:31:52.431802 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:52.431756 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 21:31:52.432016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:52.431986 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 21:31:52.652405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:52.652364 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 21:31:52.652599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:52.652431 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 21:31:52.724350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:52.721028 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:31:52.725077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:52.725038 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 21:31:52.770650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:52.770580 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" event={"ID":"2cd1ac49-8c79-4ac3-bc18-f385b79299d9","Type":"ContainerStarted","Data":"a08e35bdc4ab2978f739994b202426d2480ba8e6288c4dbc9d3120e8a90a1956"} Oct 31 21:31:53.426652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.426615 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 21:31:53.427080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.427044 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 21:31:53.701483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.663918 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 21:31:53.701483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.663983 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 21:31:53.727381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.719156 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:31:53.727381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.719231 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 21:31:53.759236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.759155 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q"] Oct 31 21:31:53.767094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.767067 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.769379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.769344 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q"] Oct 31 21:31:53.904423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.902394 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.904423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.902459 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.904423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.902530 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.904423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.902563 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.904423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.902650 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.904423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.902733 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4lp\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-kube-api-access-zz4lp\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.904423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.902784 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.904423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.902927 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.904423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.902954 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:53.964103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.963682 5188 generic.go:356] "Generic (PLEG): container finished" podID="2cd1ac49-8c79-4ac3-bc18-f385b79299d9" containerID="7e18d40d5c93c802e1b7954b5839179d1204800199196935453a0358da4669c7" exitCode=0 Oct 31 21:31:53.964103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:53.963805 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" event={"ID":"2cd1ac49-8c79-4ac3-bc18-f385b79299d9","Type":"ContainerDied","Data":"7e18d40d5c93c802e1b7954b5839179d1204800199196935453a0358da4669c7"} Oct 31 21:31:54.026355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.024884 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.026355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.024942 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.026355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.024993 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.026355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.025023 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.026355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.025075 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.026355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.025108 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.026355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.025220 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.026355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.025297 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4lp\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-kube-api-access-zz4lp\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.026355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.025365 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.053348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.050004 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.053348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.050296 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.053348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.050570 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.053348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.050773 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.058008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.057967 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.119346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.119061 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.157752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.157712 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.158461 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.158435 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.168853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.168818 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4lp\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-kube-api-access-zz4lp\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.262431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.260223 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:54.665171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.665008 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 21:31:54.665171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.665083 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 21:31:54.707899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.706064 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:54.707899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.707655 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:54.724588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.724098 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:31:54.724588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.724164 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 21:31:54.753363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.737502 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:31:54.997369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:54.997297 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q"] Oct 31 21:31:55.058360 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:31:55.052864 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ad8326_d6e9_4750_ac21_bb1d9c35ca10.slice/crio-3a0f300312732d7f225402140eac10eab45e1c3835ddbfb0e0a09452b35eabb3 WatchSource:0}: Error finding container 3a0f300312732d7f225402140eac10eab45e1c3835ddbfb0e0a09452b35eabb3: Status 404 returned error can't find the container with id 3a0f300312732d7f225402140eac10eab45e1c3835ddbfb0e0a09452b35eabb3 Oct 31 21:31:55.058360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.057946 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:55.058360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.058035 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:55.058360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.058081 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:31:55.521806 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.521724 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:55.627015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.626967 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-openshift-ca\") pod \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " Oct 31 21:31:55.627395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.627369 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-xks-ca\") pod \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " Oct 31 21:31:55.627601 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.627575 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-kube-api-access-mf64l\" (UniqueName: \"kubernetes.io/projected/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-kube-api-access-mf64l\") pod \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\" (UID: \"2cd1ac49-8c79-4ac3-bc18-f385b79299d9\") " Oct 31 21:31:55.641047 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.641001 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-openshift-ca" pod "2cd1ac49-8c79-4ac3-bc18-f385b79299d9" (UID: "2cd1ac49-8c79-4ac3-bc18-f385b79299d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:31:55.669272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.662120 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-kube-api-access-mf64l" pod "2cd1ac49-8c79-4ac3-bc18-f385b79299d9" (UID: "2cd1ac49-8c79-4ac3-bc18-f385b79299d9"). InnerVolumeSpecName "kube-api-access-mf64l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:55.669272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.663845 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-xks-ca" pod "2cd1ac49-8c79-4ac3-bc18-f385b79299d9" (UID: "2cd1ac49-8c79-4ac3-bc18-f385b79299d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:55.669272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.664487 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 21:31:55.669272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.664554 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 21:31:55.756760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.756707 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:55.756760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.756752 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf64l\" (UniqueName: \"kubernetes.io/projected/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-kube-api-access-mf64l\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:55.756760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.756767 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd1ac49-8c79-4ac3-bc18-f385b79299d9-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:31:55.757422 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.756863 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:31:55.757422 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:55.756909 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 21:31:56.001345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.001189 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" event={"ID":"62ad8326-d6e9-4750-ac21-bb1d9c35ca10","Type":"ContainerStarted","Data":"0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495"} Oct 31 21:31:56.001345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.001302 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" event={"ID":"62ad8326-d6e9-4750-ac21-bb1d9c35ca10","Type":"ContainerStarted","Data":"3a0f300312732d7f225402140eac10eab45e1c3835ddbfb0e0a09452b35eabb3"} Oct 31 21:31:56.004641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.004614 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" Oct 31 21:31:56.004999 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.004859 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc" event={"ID":"2cd1ac49-8c79-4ac3-bc18-f385b79299d9","Type":"ContainerDied","Data":"a08e35bdc4ab2978f739994b202426d2480ba8e6288c4dbc9d3120e8a90a1956"} Oct 31 21:31:56.005142 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.005129 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a08e35bdc4ab2978f739994b202426d2480ba8e6288c4dbc9d3120e8a90a1956" Oct 31 21:31:56.047346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.047099 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" podStartSLOduration=3.04708237 podStartE2EDuration="3.04708237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:56.045096041 +0000 UTC m=+962.738053437" watchObservedRunningTime="2026-10-31 21:31:56.04708237 +0000 UTC m=+962.740039764" Oct 31 21:31:56.261704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.261605 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:31:56.265623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.265577 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 21:31:56.265850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.265652 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 21:31:56.653282 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.653234 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 31 21:31:56.653580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.653536 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 31 21:31:56.724349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.717599 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:31:56.724349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:56.717679 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 31 21:31:57.261348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.261185 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 21:31:57.261348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.261256 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 21:31:57.314348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.313597 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s"] Oct 31 21:31:57.324354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.314630 5188 cpu_manager.go:414] "containerMap: removing container" podUID="2cd1ac49-8c79-4ac3-bc18-f385b79299d9" containerName="revoke-keys" Oct 31 21:31:57.324354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.314656 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd1ac49-8c79-4ac3-bc18-f385b79299d9" containerName="revoke-keys" Oct 31 21:31:57.324354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.314832 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd1ac49-8c79-4ac3-bc18-f385b79299d9" containerName="revoke-keys" Oct 31 21:31:57.336617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.336584 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s"] Oct 31 21:31:57.344408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.336955 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.364532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.364501 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:31:57.367279 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.365474 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-999edc59-bks2s\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.369177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.369149 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-999edc59-bks2s\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.369695 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.369668 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfpt\" (UniqueName: \"kubernetes.io/projected/75599e2d-53bd-4bd8-bc55-85ececa49702-kube-api-access-8lfpt\") pod \"maas-api-revoke-sub-999edc59-bks2s\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.370313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.369981 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:31:57.370503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.370045 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:31:57.370628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.369556 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:31:57.471239 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.471198 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-999edc59-bks2s\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.471467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.471366 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-999edc59-bks2s\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.471467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.471445 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfpt\" (UniqueName: \"kubernetes.io/projected/75599e2d-53bd-4bd8-bc55-85ececa49702-kube-api-access-8lfpt\") pod \"maas-api-revoke-sub-999edc59-bks2s\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.495349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.472686 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-999edc59-bks2s\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.533415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.530903 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-xks-ca\") pod \"maas-api-revoke-sub-999edc59-bks2s\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.533415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.531179 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfpt\" (UniqueName: \"kubernetes.io/projected/75599e2d-53bd-4bd8-bc55-85ececa49702-kube-api-access-8lfpt\") pod \"maas-api-revoke-sub-999edc59-bks2s\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:57.720771 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.720731 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:57.722065 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.722034 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:57.723254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.723228 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:31:57.736983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.736953 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:57.737774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.737682 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:57.741021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.739605 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:31:57.755182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:57.755139 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:31:58.090408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:58.089163 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s"] Oct 31 21:31:58.261516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:58.261480 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 21:31:58.261950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:58.261538 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 21:31:59.067075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:59.067038 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" event={"ID":"75599e2d-53bd-4bd8-bc55-85ececa49702","Type":"ContainerStarted","Data":"bef15e05841a0886c790269932f0693fc6d42a22a2a408dd449216e4e715fae8"} Oct 31 21:31:59.067264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:59.067102 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" event={"ID":"75599e2d-53bd-4bd8-bc55-85ececa49702","Type":"ContainerStarted","Data":"6dfc4596cf22042c1bd9bcb8c2fb953ae86ecafd8fc6de2262718887dcbf0add"} Oct 31 21:31:59.261296 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:59.261259 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 21:31:59.261537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:31:59.261318 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 21:32:00.093345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.093288 5188 generic.go:356] "Generic (PLEG): container finished" podID="75599e2d-53bd-4bd8-bc55-85ececa49702" containerID="bef15e05841a0886c790269932f0693fc6d42a22a2a408dd449216e4e715fae8" exitCode=0 Oct 31 21:32:00.093779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.093750 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" event={"ID":"75599e2d-53bd-4bd8-bc55-85ececa49702","Type":"ContainerDied","Data":"bef15e05841a0886c790269932f0693fc6d42a22a2a408dd449216e4e715fae8"} Oct 31 21:32:00.261747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.261706 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 21:32:00.262175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.261762 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 21:32:00.655913 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.655872 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m"] Oct 31 21:32:00.660750 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.660719 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:00.715607 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.715553 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m"] Oct 31 21:32:00.748905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.748863 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67kw4\" (UniqueName: \"kubernetes.io/projected/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-kube-api-access-67kw4\") pod \"maas-api-revoke-sub-9c6edead-95r5m\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:00.749067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.748917 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c6edead-95r5m\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:00.749067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.749037 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c6edead-95r5m\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:00.850466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.850426 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67kw4\" (UniqueName: \"kubernetes.io/projected/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-kube-api-access-67kw4\") pod \"maas-api-revoke-sub-9c6edead-95r5m\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:00.850699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.850480 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c6edead-95r5m\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:00.850699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.850650 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c6edead-95r5m\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:00.851247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.851215 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c6edead-95r5m\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:00.853292 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.853271 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c6edead-95r5m\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:00.910253 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:00.910158 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67kw4\" (UniqueName: \"kubernetes.io/projected/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-kube-api-access-67kw4\") pod \"maas-api-revoke-sub-9c6edead-95r5m\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:01.027294 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.027255 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:01.261886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.261851 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 21:32:01.262268 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.261921 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 21:32:01.429032 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.428946 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m"] Oct 31 21:32:01.429965 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:32:01.429418 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752a24f7_0fe5_4ffe_9a8e_c590ed2500f5.slice/crio-a234cc1fd3f73244469ad49e43ae8284391a5fcf9086b6a6b212ae64b512d768 WatchSource:0}: Error finding container a234cc1fd3f73244469ad49e43ae8284391a5fcf9086b6a6b212ae64b512d768: Status 404 returned error can't find the container with id a234cc1fd3f73244469ad49e43ae8284391a5fcf9086b6a6b212ae64b512d768 Oct 31 21:32:01.452682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.452653 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:32:01.457991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.457962 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-openshift-ca\") pod \"75599e2d-53bd-4bd8-bc55-85ececa49702\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " Oct 31 21:32:01.458170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.458160 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-xks-ca\") pod \"75599e2d-53bd-4bd8-bc55-85ececa49702\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " Oct 31 21:32:01.458250 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.458184 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75599e2d-53bd-4bd8-bc55-85ececa49702-kube-api-access-8lfpt\" (UniqueName: \"kubernetes.io/projected/75599e2d-53bd-4bd8-bc55-85ececa49702-kube-api-access-8lfpt\") pod \"75599e2d-53bd-4bd8-bc55-85ececa49702\" (UID: \"75599e2d-53bd-4bd8-bc55-85ececa49702\") " Oct 31 21:32:01.458497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.458471 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-openshift-ca" pod "75599e2d-53bd-4bd8-bc55-85ececa49702" (UID: "75599e2d-53bd-4bd8-bc55-85ececa49702"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:32:01.458711 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.458692 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:01.460332 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.460291 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-xks-ca" pod "75599e2d-53bd-4bd8-bc55-85ececa49702" (UID: "75599e2d-53bd-4bd8-bc55-85ececa49702"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:32:01.460440 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.460417 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75599e2d-53bd-4bd8-bc55-85ececa49702-kube-api-access-8lfpt" pod "75599e2d-53bd-4bd8-bc55-85ececa49702" (UID: "75599e2d-53bd-4bd8-bc55-85ececa49702"). InnerVolumeSpecName "kube-api-access-8lfpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:32:01.559747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.559706 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75599e2d-53bd-4bd8-bc55-85ececa49702-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:01.559747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:01.559743 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lfpt\" (UniqueName: \"kubernetes.io/projected/75599e2d-53bd-4bd8-bc55-85ececa49702-kube-api-access-8lfpt\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:02.128033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:02.127991 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" Oct 31 21:32:02.128033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:02.127997 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s" event={"ID":"75599e2d-53bd-4bd8-bc55-85ececa49702","Type":"ContainerDied","Data":"6dfc4596cf22042c1bd9bcb8c2fb953ae86ecafd8fc6de2262718887dcbf0add"} Oct 31 21:32:02.128355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:02.128073 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dfc4596cf22042c1bd9bcb8c2fb953ae86ecafd8fc6de2262718887dcbf0add" Oct 31 21:32:02.130414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:02.130378 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" event={"ID":"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5","Type":"ContainerStarted","Data":"7bbfa9c87e6527f3268d0f031b42f34eeb86c4ae1d867be483dd7344a458936a"} Oct 31 21:32:02.130557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:02.130443 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" event={"ID":"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5","Type":"ContainerStarted","Data":"a234cc1fd3f73244469ad49e43ae8284391a5fcf9086b6a6b212ae64b512d768"} Oct 31 21:32:02.168208 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:02.168138 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" podStartSLOduration=2.168117744 podStartE2EDuration="2.168117744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:32:02.167522538 +0000 UTC m=+968.860479930" watchObservedRunningTime="2026-10-31 21:32:02.168117744 +0000 UTC m=+968.861075142" Oct 31 21:32:02.261747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:02.261711 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 31 21:32:02.261958 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:02.261775 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 31 21:32:02.544522 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:32:02.544415 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:32:03.142272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:03.142232 5188 generic.go:356] "Generic (PLEG): container finished" podID="752a24f7-0fe5-4ffe-9a8e-c590ed2500f5" containerID="7bbfa9c87e6527f3268d0f031b42f34eeb86c4ae1d867be483dd7344a458936a" exitCode=0 Oct 31 21:32:03.142531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:03.142371 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" event={"ID":"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5","Type":"ContainerDied","Data":"7bbfa9c87e6527f3268d0f031b42f34eeb86c4ae1d867be483dd7344a458936a"} Oct 31 21:32:03.266092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:03.266056 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:32:03.266543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:03.266363 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:32:03.267073 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:03.267056 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:32:05.335701 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.301394 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 31 21:32:05.732347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.731444 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:05.766349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.745118 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-openshift-ca\") pod \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " Oct 31 21:32:05.766349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.745182 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-kube-api-access-67kw4\" (UniqueName: \"kubernetes.io/projected/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-kube-api-access-67kw4\") pod \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " Oct 31 21:32:05.766349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.745425 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-xks-ca\") pod \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\" (UID: \"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5\") " Oct 31 21:32:05.766349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.750562 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-openshift-ca" pod "752a24f7-0fe5-4ffe-9a8e-c590ed2500f5" (UID: "752a24f7-0fe5-4ffe-9a8e-c590ed2500f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:32:05.766349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.756373 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-kube-api-access-67kw4" pod "752a24f7-0fe5-4ffe-9a8e-c590ed2500f5" (UID: "752a24f7-0fe5-4ffe-9a8e-c590ed2500f5"). InnerVolumeSpecName "kube-api-access-67kw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:32:05.766349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.757238 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-xks-ca" pod "752a24f7-0fe5-4ffe-9a8e-c590ed2500f5" (UID: "752a24f7-0fe5-4ffe-9a8e-c590ed2500f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:32:05.859648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.853426 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:05.859648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.853471 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67kw4\" (UniqueName: \"kubernetes.io/projected/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-kube-api-access-67kw4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:05.859648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:05.853492 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:06.306486 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:06.306424 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:06.307252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:06.307225 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:32:06.337465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:06.337416 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 21:32:06.341447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:06.341407 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:32:06.809346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:06.809042 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" event={"ID":"752a24f7-0fe5-4ffe-9a8e-c590ed2500f5","Type":"ContainerDied","Data":"a234cc1fd3f73244469ad49e43ae8284391a5fcf9086b6a6b212ae64b512d768"} Oct 31 21:32:06.809346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:06.809111 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a234cc1fd3f73244469ad49e43ae8284391a5fcf9086b6a6b212ae64b512d768" Oct 31 21:32:06.809346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:06.809186 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m" Oct 31 21:32:06.896350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:06.872152 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:32:07.322483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:07.322428 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 31 21:32:07.488401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:07.447725 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:32:07.488401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:07.448179 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 21:32:07.488401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:07.448628 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:32:07.488401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:07.448719 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:32:07.934782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:07.934731 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:32:07.969538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:07.969472 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:07.969810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:07.969778 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:08.420276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:08.420228 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 31 21:32:08.452502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:08.452449 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:08.501530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:08.501472 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:08.562755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:08.562693 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:08.852958 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:08.852852 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:08.932053 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:32:08.929814 5188 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.422s" Oct 31 21:32:08.932712 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:08.932689 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:32:08.940575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:08.940495 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:09.189427 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.189379 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq"] Oct 31 21:32:09.220352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.190640 5188 cpu_manager.go:414] "containerMap: removing container" podUID="75599e2d-53bd-4bd8-bc55-85ececa49702" containerName="revoke-keys" Oct 31 21:32:09.223654 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.223607 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75599e2d-53bd-4bd8-bc55-85ececa49702" containerName="revoke-keys" Oct 31 21:32:09.247476 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.247425 5188 cpu_manager.go:414] "containerMap: removing container" podUID="752a24f7-0fe5-4ffe-9a8e-c590ed2500f5" containerName="revoke-keys" Oct 31 21:32:09.247730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.247701 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752a24f7-0fe5-4ffe-9a8e-c590ed2500f5" containerName="revoke-keys" Oct 31 21:32:09.248211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.248193 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="75599e2d-53bd-4bd8-bc55-85ececa49702" containerName="revoke-keys" Oct 31 21:32:09.248383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.248368 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="752a24f7-0fe5-4ffe-9a8e-c590ed2500f5" containerName="revoke-keys" Oct 31 21:32:09.352942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.352907 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.455572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.455463 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 21:32:09.460438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.460403 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54713ff3-ftbmq\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.460691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.460668 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jpb\" (UniqueName: \"kubernetes.io/projected/d0205e11-1145-42d4-952d-66e7b58e8416-kube-api-access-68jpb\") pod \"maas-api-revoke-sub-54713ff3-ftbmq\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.460819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.460804 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54713ff3-ftbmq\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.487363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.464277 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:32:09.487363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.464391 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:32:09.487363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.464475 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:32:09.487363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.464580 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:32:09.514553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.506422 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:32:09.574350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.560367 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq"] Oct 31 21:32:09.574350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.561785 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54713ff3-ftbmq\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.574350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.561823 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jpb\" (UniqueName: \"kubernetes.io/projected/d0205e11-1145-42d4-952d-66e7b58e8416-kube-api-access-68jpb\") pod \"maas-api-revoke-sub-54713ff3-ftbmq\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.574350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.561858 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54713ff3-ftbmq\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.574350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.564178 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54713ff3-ftbmq\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.621656 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.598079 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54713ff3-ftbmq\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.671381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.636207 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jpb\" (UniqueName: \"kubernetes.io/projected/d0205e11-1145-42d4-952d-66e7b58e8416-kube-api-access-68jpb\") pod \"maas-api-revoke-sub-54713ff3-ftbmq\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:09.749044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.748945 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:32:09.798756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:09.798582 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:10.043412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:10.040449 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:32:10.083512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:10.083467 5188 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:10.083712 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:10.083539 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:10.178347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:10.177550 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:32:10.291218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:10.287651 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:32:10.345469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:10.345149 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:32:10.668377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:10.668344 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq"] Oct 31 21:32:10.668873 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:32:10.668846 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0205e11_1145_42d4_952d_66e7b58e8416.slice/crio-94faaf14d90bbf8e7fb78c90d0902833ac5cf1464765e3c57d1db5bd919f63f1 WatchSource:0}: Error finding container 94faaf14d90bbf8e7fb78c90d0902833ac5cf1464765e3c57d1db5bd919f63f1: Status 404 returned error can't find the container with id 94faaf14d90bbf8e7fb78c90d0902833ac5cf1464765e3c57d1db5bd919f63f1 Oct 31 21:32:11.150161 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:11.150118 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" event={"ID":"d0205e11-1145-42d4-952d-66e7b58e8416","Type":"ContainerStarted","Data":"0e5091c44fde7b177b663bdeb9c0a2b86b96fb4e2473b4c44f81c879d28915c9"} Oct 31 21:32:11.150409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:11.150209 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" event={"ID":"d0205e11-1145-42d4-952d-66e7b58e8416","Type":"ContainerStarted","Data":"94faaf14d90bbf8e7fb78c90d0902833ac5cf1464765e3c57d1db5bd919f63f1"} Oct 31 21:32:11.189730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:11.189663 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" podStartSLOduration=3.189646729 podStartE2EDuration="3.189646729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:32:11.185398276 +0000 UTC m=+977.878355666" watchObservedRunningTime="2026-10-31 21:32:11.189646729 +0000 UTC m=+977.882604121" Oct 31 21:32:12.160973 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:12.160933 5188 generic.go:356] "Generic (PLEG): container finished" podID="d0205e11-1145-42d4-952d-66e7b58e8416" containerID="0e5091c44fde7b177b663bdeb9c0a2b86b96fb4e2473b4c44f81c879d28915c9" exitCode=0 Oct 31 21:32:12.161451 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:12.161024 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" event={"ID":"d0205e11-1145-42d4-952d-66e7b58e8416","Type":"ContainerDied","Data":"0e5091c44fde7b177b663bdeb9c0a2b86b96fb4e2473b4c44f81c879d28915c9"} Oct 31 21:32:13.355015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.354978 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:13.445838 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.445739 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0205e11-1145-42d4-952d-66e7b58e8416-kube-api-access-68jpb\" (UniqueName: \"kubernetes.io/projected/d0205e11-1145-42d4-952d-66e7b58e8416-kube-api-access-68jpb\") pod \"d0205e11-1145-42d4-952d-66e7b58e8416\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " Oct 31 21:32:13.445838 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.445803 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-openshift-ca\") pod \"d0205e11-1145-42d4-952d-66e7b58e8416\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " Oct 31 21:32:13.446206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.445992 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-xks-ca\") pod \"d0205e11-1145-42d4-952d-66e7b58e8416\" (UID: \"d0205e11-1145-42d4-952d-66e7b58e8416\") " Oct 31 21:32:13.446504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.446292 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-openshift-ca" pod "d0205e11-1145-42d4-952d-66e7b58e8416" (UID: "d0205e11-1145-42d4-952d-66e7b58e8416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:32:13.446937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.446909 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:13.448376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.448310 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-xks-ca" pod "d0205e11-1145-42d4-952d-66e7b58e8416" (UID: "d0205e11-1145-42d4-952d-66e7b58e8416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:32:13.448649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.448630 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0205e11-1145-42d4-952d-66e7b58e8416-kube-api-access-68jpb" pod "d0205e11-1145-42d4-952d-66e7b58e8416" (UID: "d0205e11-1145-42d4-952d-66e7b58e8416"). InnerVolumeSpecName "kube-api-access-68jpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:32:13.548517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.548480 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0205e11-1145-42d4-952d-66e7b58e8416-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:13.548517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:13.548512 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68jpb\" (UniqueName: \"kubernetes.io/projected/d0205e11-1145-42d4-952d-66e7b58e8416-kube-api-access-68jpb\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:14.179530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.179498 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" Oct 31 21:32:14.179772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.179492 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq" event={"ID":"d0205e11-1145-42d4-952d-66e7b58e8416","Type":"ContainerDied","Data":"94faaf14d90bbf8e7fb78c90d0902833ac5cf1464765e3c57d1db5bd919f63f1"} Oct 31 21:32:14.179772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.179651 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94faaf14d90bbf8e7fb78c90d0902833ac5cf1464765e3c57d1db5bd919f63f1" Oct 31 21:32:14.941901 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.941855 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2"] Oct 31 21:32:14.942524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.942506 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d0205e11-1145-42d4-952d-66e7b58e8416" containerName="revoke-keys" Oct 31 21:32:14.942567 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.942524 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0205e11-1145-42d4-952d-66e7b58e8416" containerName="revoke-keys" Oct 31 21:32:14.942653 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.942643 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0205e11-1145-42d4-952d-66e7b58e8416" containerName="revoke-keys" Oct 31 21:32:14.946605 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.946583 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:14.966507 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.966479 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2"] Oct 31 21:32:14.970143 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.970105 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:32:14.970143 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.970115 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:32:14.970361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.970143 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:32:14.970361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:14.970147 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:32:15.067805 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.067762 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d3859a-pd7h2\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.068023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.067942 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mj2z\" (UniqueName: \"kubernetes.io/projected/11a8d24e-418d-4e8d-86db-2e3f683dc435-kube-api-access-8mj2z\") pod \"maas-api-revoke-sub-08d3859a-pd7h2\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.068169 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.068147 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d3859a-pd7h2\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.169283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.169244 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mj2z\" (UniqueName: \"kubernetes.io/projected/11a8d24e-418d-4e8d-86db-2e3f683dc435-kube-api-access-8mj2z\") pod \"maas-api-revoke-sub-08d3859a-pd7h2\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.169500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.169400 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d3859a-pd7h2\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.169500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.169444 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d3859a-pd7h2\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.170156 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.170119 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d3859a-pd7h2\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.171827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.171805 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d3859a-pd7h2\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.191136 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.191092 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mj2z\" (UniqueName: \"kubernetes.io/projected/11a8d24e-418d-4e8d-86db-2e3f683dc435-kube-api-access-8mj2z\") pod \"maas-api-revoke-sub-08d3859a-pd7h2\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.262224 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.262135 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:15.574099 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:32:15.574063 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a8d24e_418d_4e8d_86db_2e3f683dc435.slice/crio-ee2b66d942b680bbec067d370902fbcfb27d8bff977190c1b34d3b3ba1715da7 WatchSource:0}: Error finding container ee2b66d942b680bbec067d370902fbcfb27d8bff977190c1b34d3b3ba1715da7: Status 404 returned error can't find the container with id ee2b66d942b680bbec067d370902fbcfb27d8bff977190c1b34d3b3ba1715da7 Oct 31 21:32:15.574301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:15.574271 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2"] Oct 31 21:32:16.201760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:16.201660 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" event={"ID":"11a8d24e-418d-4e8d-86db-2e3f683dc435","Type":"ContainerStarted","Data":"1d864ea87c9f1ac8824f2da6b6c070458581693a7cb443c0a09a7bdb3b6e9e30"} Oct 31 21:32:16.201760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:16.201727 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" event={"ID":"11a8d24e-418d-4e8d-86db-2e3f683dc435","Type":"ContainerStarted","Data":"ee2b66d942b680bbec067d370902fbcfb27d8bff977190c1b34d3b3ba1715da7"} Oct 31 21:32:16.241420 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:16.241362 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" podStartSLOduration=2.241345757 podStartE2EDuration="2.241345757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:32:16.237065608 +0000 UTC m=+982.930023002" watchObservedRunningTime="2026-10-31 21:32:16.241345757 +0000 UTC m=+982.934303192" Oct 31 21:32:17.215862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:17.215826 5188 generic.go:356] "Generic (PLEG): container finished" podID="11a8d24e-418d-4e8d-86db-2e3f683dc435" containerID="1d864ea87c9f1ac8824f2da6b6c070458581693a7cb443c0a09a7bdb3b6e9e30" exitCode=0 Oct 31 21:32:17.216425 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:17.215920 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" event={"ID":"11a8d24e-418d-4e8d-86db-2e3f683dc435","Type":"ContainerDied","Data":"1d864ea87c9f1ac8824f2da6b6c070458581693a7cb443c0a09a7bdb3b6e9e30"} Oct 31 21:32:18.400811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.400774 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:18.520963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.520869 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-openshift-ca\") pod \"11a8d24e-418d-4e8d-86db-2e3f683dc435\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " Oct 31 21:32:18.520963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.520952 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a8d24e-418d-4e8d-86db-2e3f683dc435-kube-api-access-8mj2z\" (UniqueName: \"kubernetes.io/projected/11a8d24e-418d-4e8d-86db-2e3f683dc435-kube-api-access-8mj2z\") pod \"11a8d24e-418d-4e8d-86db-2e3f683dc435\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " Oct 31 21:32:18.521217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.521059 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-xks-ca\") pod \"11a8d24e-418d-4e8d-86db-2e3f683dc435\" (UID: \"11a8d24e-418d-4e8d-86db-2e3f683dc435\") " Oct 31 21:32:18.521366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.521339 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-openshift-ca" pod "11a8d24e-418d-4e8d-86db-2e3f683dc435" (UID: "11a8d24e-418d-4e8d-86db-2e3f683dc435"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:32:18.521904 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.521874 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:18.523541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.523504 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-xks-ca" pod "11a8d24e-418d-4e8d-86db-2e3f683dc435" (UID: "11a8d24e-418d-4e8d-86db-2e3f683dc435"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:32:18.523985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.523963 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a8d24e-418d-4e8d-86db-2e3f683dc435-kube-api-access-8mj2z" pod "11a8d24e-418d-4e8d-86db-2e3f683dc435" (UID: "11a8d24e-418d-4e8d-86db-2e3f683dc435"). InnerVolumeSpecName "kube-api-access-8mj2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:32:18.622991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.622944 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mj2z\" (UniqueName: \"kubernetes.io/projected/11a8d24e-418d-4e8d-86db-2e3f683dc435-kube-api-access-8mj2z\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:18.622991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:18.622983 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11a8d24e-418d-4e8d-86db-2e3f683dc435-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:19.236286 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.236252 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" Oct 31 21:32:19.236489 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.236281 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2" event={"ID":"11a8d24e-418d-4e8d-86db-2e3f683dc435","Type":"ContainerDied","Data":"ee2b66d942b680bbec067d370902fbcfb27d8bff977190c1b34d3b3ba1715da7"} Oct 31 21:32:19.236489 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.236361 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee2b66d942b680bbec067d370902fbcfb27d8bff977190c1b34d3b3ba1715da7" Oct 31 21:32:19.262188 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.262149 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm"] Oct 31 21:32:19.262827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.262811 5188 cpu_manager.go:414] "containerMap: removing container" podUID="11a8d24e-418d-4e8d-86db-2e3f683dc435" containerName="revoke-keys" Oct 31 21:32:19.262879 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.262829 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a8d24e-418d-4e8d-86db-2e3f683dc435" containerName="revoke-keys" Oct 31 21:32:19.262964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.262955 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a8d24e-418d-4e8d-86db-2e3f683dc435" containerName="revoke-keys" Oct 31 21:32:19.266935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.266910 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.282846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.282808 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-cc45a7-openshift-default-dockercfg-dbkx4\"" Oct 31 21:32:19.296495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.294406 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm"] Oct 31 21:32:19.434012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.433965 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.434482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.434056 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80080a1d-b279-4bab-a2da-48c1cd251701-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.434482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.434152 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.434482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.434200 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.434482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.434292 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80080a1d-b279-4bab-a2da-48c1cd251701-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.434482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.434409 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.434650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.434555 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.434650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.434620 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vjp6\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-kube-api-access-7vjp6\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.434711 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.434681 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.535867 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.535771 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.535867 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.535831 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vjp6\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-kube-api-access-7vjp6\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.535882 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.535904 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.535925 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80080a1d-b279-4bab-a2da-48c1cd251701-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.536071 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.536103 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536419 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.536255 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80080a1d-b279-4bab-a2da-48c1cd251701-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536419 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.536340 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536520 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.536262 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.536513 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.536721 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.536750 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.536871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.536843 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80080a1d-b279-4bab-a2da-48c1cd251701-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.539045 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.539014 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80080a1d-b279-4bab-a2da-48c1cd251701-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.539169 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.539059 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.558280 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.558251 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.558467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.558427 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vjp6\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-kube-api-access-7vjp6\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.583189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.583131 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:19.922613 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:32:19.922578 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80080a1d_b279_4bab_a2da_48c1cd251701.slice/crio-85932a20692a889701ee3329b5f37c005d50cdce0bb3fbf442ed9cbc9f9ebaa1 WatchSource:0}: Error finding container 85932a20692a889701ee3329b5f37c005d50cdce0bb3fbf442ed9cbc9f9ebaa1: Status 404 returned error can't find the container with id 85932a20692a889701ee3329b5f37c005d50cdce0bb3fbf442ed9cbc9f9ebaa1 Oct 31 21:32:19.923757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.923726 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm"] Oct 31 21:32:19.925283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.925240 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:32:19.925411 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.925365 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:32:19.925456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:19.925414 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:32:20.248358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:20.248237 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" event={"ID":"80080a1d-b279-4bab-a2da-48c1cd251701","Type":"ContainerStarted","Data":"79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9"} Oct 31 21:32:20.248358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:20.248312 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" event={"ID":"80080a1d-b279-4bab-a2da-48c1cd251701","Type":"ContainerStarted","Data":"85932a20692a889701ee3329b5f37c005d50cdce0bb3fbf442ed9cbc9f9ebaa1"} Oct 31 21:32:20.308984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:20.308909 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podStartSLOduration=1.308886827 podStartE2EDuration="1.308886827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:32:20.308091002 +0000 UTC m=+987.001048394" watchObservedRunningTime="2026-10-31 21:32:20.308886827 +0000 UTC m=+987.001844221" Oct 31 21:32:20.584511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:20.584407 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:20.589040 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:20.588666 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:20.589040 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:20.588731 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:21.600346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:21.592809 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:21.600346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:21.592872 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:22.583767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:22.583725 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:22.584035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:22.584010 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:23.617151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:23.603298 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:23.617841 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:23.617799 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:24.041303 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.041200 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:24.238352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.235388 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9"] Oct 31 21:32:24.280355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.277531 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:24.347443 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.345756 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 31 21:32:24.387542 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.383290 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9"] Oct 31 21:32:24.387845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.387493 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.413742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.413712 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:32:24.414043 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.414008 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:32:24.414299 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.414282 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:32:24.414651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.414570 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:32:24.462791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.462744 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:24.527910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.527856 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:24.569031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.568990 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f8127c6-px9s9\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.569522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.569499 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9sgk\" (UniqueName: \"kubernetes.io/projected/0289b214-3ddd-4788-9efe-ae79885c6003-kube-api-access-g9sgk\") pod \"maas-api-revoke-sub-6f8127c6-px9s9\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.569711 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.569697 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f8127c6-px9s9\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.588964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.588925 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:24.589215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.589180 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:24.675346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.670955 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f8127c6-px9s9\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.675346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.671159 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9sgk\" (UniqueName: \"kubernetes.io/projected/0289b214-3ddd-4788-9efe-ae79885c6003-kube-api-access-g9sgk\") pod \"maas-api-revoke-sub-6f8127c6-px9s9\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.675346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.671204 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f8127c6-px9s9\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.675346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.672261 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f8127c6-px9s9\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.698648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.689981 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f8127c6-px9s9\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.737936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.737862 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9sgk\" (UniqueName: \"kubernetes.io/projected/0289b214-3ddd-4788-9efe-ae79885c6003-kube-api-access-g9sgk\") pod \"maas-api-revoke-sub-6f8127c6-px9s9\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:24.838776 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:24.838667 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:25.045363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.043473 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:32:25.045363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.044447 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:32:25.374405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.349547 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:25.419183 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.419148 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:32:25.498350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.489422 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:25.545349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.541845 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:25.651468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.616139 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:25.651468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.616248 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:25.716348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.715595 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:25.839349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.784426 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:25.839349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:25.784489 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:26.062351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:26.028752 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:32:26.399403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:26.135430 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:26.401600 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:26.401496 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:32:26.442550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:26.442511 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:32:26.492952 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:26.492685 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:26.624351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:26.573253 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:27.300210 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:26.593159 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:27.301374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:27.301339 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:27.430452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:27.415538 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 31 21:32:27.494008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:27.493958 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:27.538349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:27.520308 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:27.661455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:27.547758 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz"] Oct 31 21:32:27.802291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:27.798204 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:27.951094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:27.910655 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:27.951094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:27.910718 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:28.142350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.126824 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f425\" (UniqueName: \"kubernetes.io/projected/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-kube-api-access-6f425\") pod \"maas-api-revoke-sub-8ce8caab-s2rkz\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.142846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.142809 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ce8caab-s2rkz\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.143071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.143052 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ce8caab-s2rkz\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.307684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.307641 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:32:28.347377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.341082 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz"] Oct 31 21:32:28.433349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.427042 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ce8caab-s2rkz\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.433349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.427144 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ce8caab-s2rkz\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.433349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.427398 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f425\" (UniqueName: \"kubernetes.io/projected/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-kube-api-access-6f425\") pod \"maas-api-revoke-sub-8ce8caab-s2rkz\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.433349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.430214 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ce8caab-s2rkz\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.433349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.430316 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:28.433349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.431578 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:32:28.438639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.438604 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ce8caab-s2rkz\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.525413 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.494182 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f425\" (UniqueName: \"kubernetes.io/projected/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-kube-api-access-6f425\") pod \"maas-api-revoke-sub-8ce8caab-s2rkz\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.525788 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.498666 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:28.624349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.576135 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:32:28.624349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.603789 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:28.624349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.603850 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:28.685349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.660006 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:28.854847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.854438 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:28.854847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.854516 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:28.961372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.959769 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:28.961372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:28.959834 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:29.222570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:29.209645 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" event={"ID":"0289b214-3ddd-4788-9efe-ae79885c6003","Type":"ContainerStarted","Data":"904a4f7837ff26ff850b4f32662d53b4581ce0b4e4828802194f206ef387b82a"} Oct 31 21:32:29.272350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:29.225415 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9"] Oct 31 21:32:29.309312 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:29.308413 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:29.437482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:29.437429 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:29.561352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:29.544404 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:29.584349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:29.576514 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:29.593251 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:29.593216 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:29.600461 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:29.600412 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:29.840082 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:29.833579 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:32:30.391095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.308608 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:30.556227 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.555425 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:30.611004 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.608197 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:30.642546 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.634378 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:30.642546 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.634436 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:30.663234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.663204 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:32:30.722351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.682543 5188 generic.go:356] "Generic (PLEG): container finished" podID="0289b214-3ddd-4788-9efe-ae79885c6003" containerID="fa8d231a45817bba9af88a90211933dc7376344b6dbd9c8d17bf945f31fc9a24" exitCode=0 Oct 31 21:32:30.723036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.722992 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" event={"ID":"0289b214-3ddd-4788-9efe-ae79885c6003","Type":"ContainerDied","Data":"fa8d231a45817bba9af88a90211933dc7376344b6dbd9c8d17bf945f31fc9a24"} Oct 31 21:32:30.736342 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.735352 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:32:30.736797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.736763 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz"] Oct 31 21:32:30.738139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:30.738107 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:32:31.595155 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:31.595112 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:31.595797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:31.595763 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:31.733972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:31.733936 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" event={"ID":"f79b16cd-9abb-42bc-950e-5c7f4bd0397f","Type":"ContainerStarted","Data":"0ae8cfb1b9e80e8f08400f6fccaaa1c3f65920b4e4aed05eac77e9a0589d189d"} Oct 31 21:32:31.734178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:31.733995 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" event={"ID":"f79b16cd-9abb-42bc-950e-5c7f4bd0397f","Type":"ContainerStarted","Data":"643395cc65251018f1dfb21c43bb0a9f29cbb7d8c067164aab063fd7025ecf78"} Oct 31 21:32:31.775352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:31.773578 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" podStartSLOduration=5.773558434 podStartE2EDuration="5.773558434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:32:31.769917693 +0000 UTC m=+998.462875085" watchObservedRunningTime="2026-10-31 21:32:31.773558434 +0000 UTC m=+998.466515828" Oct 31 21:32:31.968599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:31.968567 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:32.050663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.050620 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-openshift-ca\") pod \"0289b214-3ddd-4788-9efe-ae79885c6003\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " Oct 31 21:32:32.050865 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.050718 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0289b214-3ddd-4788-9efe-ae79885c6003-kube-api-access-g9sgk\" (UniqueName: \"kubernetes.io/projected/0289b214-3ddd-4788-9efe-ae79885c6003-kube-api-access-g9sgk\") pod \"0289b214-3ddd-4788-9efe-ae79885c6003\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " Oct 31 21:32:32.050865 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.050766 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-xks-ca\") pod \"0289b214-3ddd-4788-9efe-ae79885c6003\" (UID: \"0289b214-3ddd-4788-9efe-ae79885c6003\") " Oct 31 21:32:32.051054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.051017 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-openshift-ca" pod "0289b214-3ddd-4788-9efe-ae79885c6003" (UID: "0289b214-3ddd-4788-9efe-ae79885c6003"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:32:32.051492 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.051460 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:32.052925 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.052900 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0289b214-3ddd-4788-9efe-ae79885c6003-kube-api-access-g9sgk" pod "0289b214-3ddd-4788-9efe-ae79885c6003" (UID: "0289b214-3ddd-4788-9efe-ae79885c6003"). InnerVolumeSpecName "kube-api-access-g9sgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:32:32.053033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.052955 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-xks-ca" pod "0289b214-3ddd-4788-9efe-ae79885c6003" (UID: "0289b214-3ddd-4788-9efe-ae79885c6003"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:32:32.152498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.152461 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9sgk\" (UniqueName: \"kubernetes.io/projected/0289b214-3ddd-4788-9efe-ae79885c6003-kube-api-access-g9sgk\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:32.152498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.152494 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0289b214-3ddd-4788-9efe-ae79885c6003-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:32.546258 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:32:32.546212 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:32:32.583867 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.583825 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 31 21:32:32.584090 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.583902 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 31 21:32:32.745211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.745177 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" Oct 31 21:32:32.745727 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.745209 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9" event={"ID":"0289b214-3ddd-4788-9efe-ae79885c6003","Type":"ContainerDied","Data":"904a4f7837ff26ff850b4f32662d53b4581ce0b4e4828802194f206ef387b82a"} Oct 31 21:32:32.745727 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.745272 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="904a4f7837ff26ff850b4f32662d53b4581ce0b4e4828802194f206ef387b82a" Oct 31 21:32:32.747127 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.747099 5188 generic.go:356] "Generic (PLEG): container finished" podID="f79b16cd-9abb-42bc-950e-5c7f4bd0397f" containerID="0ae8cfb1b9e80e8f08400f6fccaaa1c3f65920b4e4aed05eac77e9a0589d189d" exitCode=0 Oct 31 21:32:32.747268 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:32.747182 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" event={"ID":"f79b16cd-9abb-42bc-950e-5c7f4bd0397f","Type":"ContainerDied","Data":"0ae8cfb1b9e80e8f08400f6fccaaa1c3f65920b4e4aed05eac77e9a0589d189d"} Oct 31 21:32:34.019481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.014637 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:34.095345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.093387 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-openshift-ca\") pod \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " Oct 31 21:32:34.095345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.093563 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-xks-ca\") pod \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " Oct 31 21:32:34.095345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.093650 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-kube-api-access-6f425\" (UniqueName: \"kubernetes.io/projected/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-kube-api-access-6f425\") pod \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\" (UID: \"f79b16cd-9abb-42bc-950e-5c7f4bd0397f\") " Oct 31 21:32:34.095345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.094262 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-openshift-ca" pod "f79b16cd-9abb-42bc-950e-5c7f4bd0397f" (UID: "f79b16cd-9abb-42bc-950e-5c7f4bd0397f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:32:34.101374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.097677 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-xks-ca" pod "f79b16cd-9abb-42bc-950e-5c7f4bd0397f" (UID: "f79b16cd-9abb-42bc-950e-5c7f4bd0397f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:32:34.101374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.099498 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-kube-api-access-6f425" pod "f79b16cd-9abb-42bc-950e-5c7f4bd0397f" (UID: "f79b16cd-9abb-42bc-950e-5c7f4bd0397f"). InnerVolumeSpecName "kube-api-access-6f425". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:32:34.196028 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.195975 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:34.196028 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.196021 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:34.196028 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.196036 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f425\" (UniqueName: \"kubernetes.io/projected/f79b16cd-9abb-42bc-950e-5c7f4bd0397f-kube-api-access-6f425\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:34.482405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.482373 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:34.482893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.482862 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:34.483895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.483876 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 21:32:34.699640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.699602 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg"] Oct 31 21:32:34.700679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.700653 5188 cpu_manager.go:414] "containerMap: removing container" podUID="0289b214-3ddd-4788-9efe-ae79885c6003" containerName="revoke-keys" Oct 31 21:32:34.700679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.700678 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0289b214-3ddd-4788-9efe-ae79885c6003" containerName="revoke-keys" Oct 31 21:32:34.700834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.700758 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f79b16cd-9abb-42bc-950e-5c7f4bd0397f" containerName="revoke-keys" Oct 31 21:32:34.700834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.700768 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79b16cd-9abb-42bc-950e-5c7f4bd0397f" containerName="revoke-keys" Oct 31 21:32:34.700950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.700936 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79b16cd-9abb-42bc-950e-5c7f4bd0397f" containerName="revoke-keys" Oct 31 21:32:34.700993 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.700963 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="0289b214-3ddd-4788-9efe-ae79885c6003" containerName="revoke-keys" Oct 31 21:32:34.705797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.705770 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:34.725013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.724973 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg"] Oct 31 21:32:34.772918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.772829 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" Oct 31 21:32:34.773410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.773232 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz" event={"ID":"f79b16cd-9abb-42bc-950e-5c7f4bd0397f","Type":"ContainerDied","Data":"643395cc65251018f1dfb21c43bb0a9f29cbb7d8c067164aab063fd7025ecf78"} Oct 31 21:32:34.774389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.773430 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643395cc65251018f1dfb21c43bb0a9f29cbb7d8c067164aab063fd7025ecf78" Oct 31 21:32:34.805725 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.805670 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8426f638-2psbg\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:34.805939 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.805743 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhlh9\" (UniqueName: \"kubernetes.io/projected/6082f947-acba-4b6e-8945-def617bcbc54-kube-api-access-vhlh9\") pod \"maas-api-revoke-sub-8426f638-2psbg\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:34.805939 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.805916 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8426f638-2psbg\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:34.907259 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.907214 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8426f638-2psbg\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:34.907491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.907300 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhlh9\" (UniqueName: \"kubernetes.io/projected/6082f947-acba-4b6e-8945-def617bcbc54-kube-api-access-vhlh9\") pod \"maas-api-revoke-sub-8426f638-2psbg\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:34.907751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.907725 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8426f638-2psbg\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:34.908363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.908337 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8426f638-2psbg\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:34.909723 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.909704 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8426f638-2psbg\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:34.929394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:34.929358 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhlh9\" (UniqueName: \"kubernetes.io/projected/6082f947-acba-4b6e-8945-def617bcbc54-kube-api-access-vhlh9\") pod \"maas-api-revoke-sub-8426f638-2psbg\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:35.025470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:35.025385 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:35.345911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:35.345877 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg"] Oct 31 21:32:35.784204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:35.784161 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" event={"ID":"6082f947-acba-4b6e-8945-def617bcbc54","Type":"ContainerStarted","Data":"d2b3c71aa85bd922e31e069cea263571dab1d0f4bf3b90c1f09fcbf99a867f44"} Oct 31 21:32:36.804614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:36.804577 5188 generic.go:356] "Generic (PLEG): container finished" podID="6082f947-acba-4b6e-8945-def617bcbc54" containerID="6a8417eea0bf8cb417678eb882badbde196dba9a377f605f03ccb981792d10b4" exitCode=0 Oct 31 21:32:36.804989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:36.804660 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" event={"ID":"6082f947-acba-4b6e-8945-def617bcbc54","Type":"ContainerDied","Data":"6a8417eea0bf8cb417678eb882badbde196dba9a377f605f03ccb981792d10b4"} Oct 31 21:32:37.987635 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:37.987610 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:38.051623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.051587 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-openshift-ca\") pod \"6082f947-acba-4b6e-8945-def617bcbc54\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " Oct 31 21:32:38.051821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.051656 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-xks-ca\") pod \"6082f947-acba-4b6e-8945-def617bcbc54\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " Oct 31 21:32:38.051821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.051702 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6082f947-acba-4b6e-8945-def617bcbc54-kube-api-access-vhlh9\" (UniqueName: \"kubernetes.io/projected/6082f947-acba-4b6e-8945-def617bcbc54-kube-api-access-vhlh9\") pod \"6082f947-acba-4b6e-8945-def617bcbc54\" (UID: \"6082f947-acba-4b6e-8945-def617bcbc54\") " Oct 31 21:32:38.052030 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.052002 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-openshift-ca" pod "6082f947-acba-4b6e-8945-def617bcbc54" (UID: "6082f947-acba-4b6e-8945-def617bcbc54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:32:38.052301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.052274 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:38.054246 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.054189 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-xks-ca" pod "6082f947-acba-4b6e-8945-def617bcbc54" (UID: "6082f947-acba-4b6e-8945-def617bcbc54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:32:38.054246 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.054199 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6082f947-acba-4b6e-8945-def617bcbc54-kube-api-access-vhlh9" pod "6082f947-acba-4b6e-8945-def617bcbc54" (UID: "6082f947-acba-4b6e-8945-def617bcbc54"). InnerVolumeSpecName "kube-api-access-vhlh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:32:38.154069 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.154035 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6082f947-acba-4b6e-8945-def617bcbc54-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:38.154069 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.154068 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhlh9\" (UniqueName: \"kubernetes.io/projected/6082f947-acba-4b6e-8945-def617bcbc54-kube-api-access-vhlh9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:38.824221 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.824177 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" event={"ID":"6082f947-acba-4b6e-8945-def617bcbc54","Type":"ContainerDied","Data":"d2b3c71aa85bd922e31e069cea263571dab1d0f4bf3b90c1f09fcbf99a867f44"} Oct 31 21:32:38.824459 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.824236 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2b3c71aa85bd922e31e069cea263571dab1d0f4bf3b90c1f09fcbf99a867f44" Oct 31 21:32:38.824459 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:38.824188 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg" Oct 31 21:32:46.561353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:46.560421 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:46.890351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:46.889561 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:47.304407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:47.303896 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:47.505349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:47.504589 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:47.565406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:47.562425 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:32:47.565406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:47.563273 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:32:47.891348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:47.890723 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:47.916350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:47.891606 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:32:48.320418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:48.318078 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.324351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:48.323541 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:32:48.543376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:48.518117 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.562938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:48.562882 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.595113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:48.584301 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:32:48.912959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:48.898126 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.329353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.329236 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.570189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.568784 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:32:49.570189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.569115 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 21:32:49.692544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.692508 5188 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:49.696497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.696461 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.774766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.773433 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592"] Oct 31 21:32:49.776175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.776142 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6082f947-acba-4b6e-8945-def617bcbc54" containerName="revoke-keys" Oct 31 21:32:49.831416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.831371 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6082f947-acba-4b6e-8945-def617bcbc54" containerName="revoke-keys" Oct 31 21:32:49.832024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.831997 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6082f947-acba-4b6e-8945-def617bcbc54" containerName="revoke-keys" Oct 31 21:32:49.857786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.857734 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:49.897406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.897362 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592"] Oct 31 21:32:49.900667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.900625 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.949425 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.924210 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:49.949844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.949808 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:49.950027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.950006 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:49.950227 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.950206 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a833011-9598-4e65-ac99-590661134d34-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:49.950549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.950527 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:49.955355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.950700 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:49.955617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.955586 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a833011-9598-4e65-ac99-590661134d34-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:49.955765 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.955745 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfpt9\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-kube-api-access-kfpt9\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:49.960489 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:49.960442 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.022349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.022212 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:32:50.062250 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.062199 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.066671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.066643 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.070462 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.070429 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a833011-9598-4e65-ac99-590661134d34-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.080488 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.080446 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfpt9\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-kube-api-access-kfpt9\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.080759 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.080743 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.084356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.080916 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.093361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.084633 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.164348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.064993 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.164348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.091063 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.164348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.091316 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.164348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.160986 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.164348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.161128 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a833011-9598-4e65-ac99-590661134d34-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.164348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.162249 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.164348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.162897 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a833011-9598-4e65-ac99-590661134d34-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.388728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.388692 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a833011-9598-4e65-ac99-590661134d34-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.512350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.490752 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.526788 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.526114 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfpt9\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-kube-api-access-kfpt9\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.526788 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.526544 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.591375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.572740 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:50.592728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.592665 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:50.617366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.613715 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:50.617842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.617768 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:50.618079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.618009 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:50.910549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.898184 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 31 21:32:50.938036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:50.913267 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:51.337549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:51.337431 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:32:51.588397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:51.583513 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:51.617354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:51.602537 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:51.920979 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:51.920928 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 31 21:32:52.303834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.295809 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592"] Oct 31 21:32:52.326404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.325978 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:32:52.326404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.326078 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:32:52.326404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.326131 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:32:52.354677 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.354633 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:32:52.355941 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.355910 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:32:52.359602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.359576 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:32:52.665567 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.599596 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:52.665567 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.617129 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 21:32:52.957905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.957810 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:32:52.965021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.960138 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:32:52.965021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:52.961028 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:32:53.294154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.294059 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" event={"ID":"7a833011-9598-4e65-ac99-590661134d34","Type":"ContainerStarted","Data":"bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8"} Oct 31 21:32:53.294439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.294409 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" event={"ID":"7a833011-9598-4e65-ac99-590661134d34","Type":"ContainerStarted","Data":"535ec8f8291176a0743ee84fdf316c6c8bd218deafe4ddfbd340e6111f48b51e"} Oct 31 21:32:53.348366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.347085 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" podStartSLOduration=4.347066005 podStartE2EDuration="4.347066005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:32:53.341822206 +0000 UTC m=+1020.034779600" watchObservedRunningTime="2026-10-31 21:32:53.347066005 +0000 UTC m=+1020.040023399" Oct 31 21:32:53.618146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.618110 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:32:53.812897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.812856 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w"] Oct 31 21:32:53.822725 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.821811 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.845895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.845856 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w"] Oct 31 21:32:53.905416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.886284 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.905416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.886400 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.905416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.886536 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvwjs\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-kube-api-access-dvwjs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.905416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.886567 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.905416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.886604 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.905416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.886636 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.905416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.886665 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90363274-a646-4f12-87f9-21439d37625c-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.905416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.886695 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.905416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.886785 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90363274-a646-4f12-87f9-21439d37625c-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.994346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.994239 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.994893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.994356 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.994893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.994483 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvwjs\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-kube-api-access-dvwjs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.994893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.994518 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.994893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.994559 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.994893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.994586 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.994893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.994613 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90363274-a646-4f12-87f9-21439d37625c-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.994893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.994642 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:53.994893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.994732 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90363274-a646-4f12-87f9-21439d37625c-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.003349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.995786 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90363274-a646-4f12-87f9-21439d37625c-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.003349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:53.996096 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.003349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.003277 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.003679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.003410 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.003799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.003769 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.005231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.005197 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.006805 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.006772 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90363274-a646-4f12-87f9-21439d37625c-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.025379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.025345 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.025742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.025720 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvwjs\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-kube-api-access-dvwjs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.058306 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.058270 5188 scope.go:122] "RemoveContainer" containerID="7ba789117b76d530614543820382103b4157db80f509a4db5563225e071409cb" Oct 31 21:32:54.069497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.069471 5188 scope.go:122] "RemoveContainer" containerID="0cf06ee2f76b24f2ee5c0dbee41da1c1e36933e0d13634bbe9518996dc30b53b" Oct 31 21:32:54.116311 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.116285 5188 scope.go:122] "RemoveContainer" containerID="d031ab1db19664299c3a50b54ce3a9fda1855f328f06f32dc840fe4b3e1ab812" Oct 31 21:32:54.127080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.127054 5188 scope.go:122] "RemoveContainer" containerID="30e246d69625b34da3f21e16781dfe6bb6d5d9f775fa13121d32e6e7ff3d2dae" Oct 31 21:32:54.137335 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.137299 5188 scope.go:122] "RemoveContainer" containerID="0dbef293373ec818c307b1e15af2938de8b2f69f8f1ea9b710420e4864382f3b" Oct 31 21:32:54.154875 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.154743 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:54.437431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.436051 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26"] Oct 31 21:32:54.442405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.442376 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.454918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.454885 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:32:54.455046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.455019 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:32:54.455155 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.455129 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:32:54.455244 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.455203 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:32:54.457984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.457959 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26"] Oct 31 21:32:54.514359 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.514295 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598d3869-h5b26\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.514550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.514515 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598d3869-h5b26\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.514715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.514699 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bklvc\" (UniqueName: \"kubernetes.io/projected/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-kube-api-access-bklvc\") pod \"maas-api-revoke-sub-598d3869-h5b26\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.537267 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:32:54.537223 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90363274_a646_4f12_87f9_21439d37625c.slice/crio-0eb1f0ebff9b71550aadb3911881d0adeb74369087a4416d01eebe6746025396 WatchSource:0}: Error finding container 0eb1f0ebff9b71550aadb3911881d0adeb74369087a4416d01eebe6746025396: Status 404 returned error can't find the container with id 0eb1f0ebff9b71550aadb3911881d0adeb74369087a4416d01eebe6746025396 Oct 31 21:32:54.540395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.540361 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:32:54.540504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.540435 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:32:54.540504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.540465 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:32:54.541843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.541819 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w"] Oct 31 21:32:54.617740 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.616975 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598d3869-h5b26\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.617740 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.617036 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598d3869-h5b26\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.617740 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.617126 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bklvc\" (UniqueName: \"kubernetes.io/projected/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-kube-api-access-bklvc\") pod \"maas-api-revoke-sub-598d3869-h5b26\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.618014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.617975 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598d3869-h5b26\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.619249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.619217 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 21:32:54.619369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.619279 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 21:32:54.619664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.619645 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598d3869-h5b26\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.638288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.638267 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bklvc\" (UniqueName: \"kubernetes.io/projected/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-kube-api-access-bklvc\") pod \"maas-api-revoke-sub-598d3869-h5b26\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:54.760664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:54.760570 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:55.118378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:55.118137 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26"] Oct 31 21:32:55.351554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:55.351496 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" event={"ID":"35627e3c-c0a8-48a0-9bfd-9732d2385ebd","Type":"ContainerStarted","Data":"f05c5219bb38621c4bb0c4de0e8b512b151bf055f7e4c0f97e772f7fdfcebe87"} Oct 31 21:32:55.354556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:55.354501 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" event={"ID":"90363274-a646-4f12-87f9-21439d37625c","Type":"ContainerStarted","Data":"d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367"} Oct 31 21:32:55.354722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:55.354579 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" event={"ID":"90363274-a646-4f12-87f9-21439d37625c","Type":"ContainerStarted","Data":"0eb1f0ebff9b71550aadb3911881d0adeb74369087a4416d01eebe6746025396"} Oct 31 21:32:55.443354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:55.443144 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" podStartSLOduration=2.443125586 podStartE2EDuration="2.443125586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:32:55.428952221 +0000 UTC m=+1022.121909616" watchObservedRunningTime="2026-10-31 21:32:55.443125586 +0000 UTC m=+1022.136082982" Oct 31 21:32:55.623153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:55.620743 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 21:32:55.623153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:55.620819 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 21:32:56.155579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:56.155543 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:32:56.157200 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:56.157163 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 21:32:56.157366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:56.157227 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 21:32:56.364367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:56.364310 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" event={"ID":"35627e3c-c0a8-48a0-9bfd-9732d2385ebd","Type":"ContainerStarted","Data":"41c21ad47e243bd4b7de8d53f12410a03c1d39a38b00ba5b870f154d93792e14"} Oct 31 21:32:56.619349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:56.618943 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 21:32:56.619349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:56.619034 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 21:32:57.156177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:57.156136 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 21:32:57.156710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:57.156196 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 21:32:57.387528 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:57.387481 5188 generic.go:356] "Generic (PLEG): container finished" podID="35627e3c-c0a8-48a0-9bfd-9732d2385ebd" containerID="41c21ad47e243bd4b7de8d53f12410a03c1d39a38b00ba5b870f154d93792e14" exitCode=0 Oct 31 21:32:57.389349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:57.389017 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" event={"ID":"35627e3c-c0a8-48a0-9bfd-9732d2385ebd","Type":"ContainerDied","Data":"41c21ad47e243bd4b7de8d53f12410a03c1d39a38b00ba5b870f154d93792e14"} Oct 31 21:32:57.641899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:57.641844 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 21:32:57.642194 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:57.642154 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 21:32:58.155075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.155039 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 21:32:58.155265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.155092 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 21:32:58.568407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.564311 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk"] Oct 31 21:32:58.591346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.585806 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk"] Oct 31 21:32:58.591346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.585929 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:58.675350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.670674 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 21:32:58.675350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.670744 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 21:32:58.762625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.759076 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc147650-l69pk\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:58.762625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.759222 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc147650-l69pk\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:58.762625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.759372 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz7n\" (UniqueName: \"kubernetes.io/projected/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-kube-api-access-zvz7n\") pod \"maas-api-revoke-sub-dc147650-l69pk\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:58.860786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.860732 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc147650-l69pk\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:58.860935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.860889 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc147650-l69pk\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:58.861018 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.861001 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz7n\" (UniqueName: \"kubernetes.io/projected/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-kube-api-access-zvz7n\") pod \"maas-api-revoke-sub-dc147650-l69pk\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:58.862999 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.862969 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc147650-l69pk\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:58.865355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.865286 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc147650-l69pk\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:58.931414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:58.930284 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz7n\" (UniqueName: \"kubernetes.io/projected/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-kube-api-access-zvz7n\") pod \"maas-api-revoke-sub-dc147650-l69pk\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:59.030935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.030524 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:59.144418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.144383 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:32:59.156086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.156047 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 21:32:59.156219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.156109 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 21:32:59.169087 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.168996 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-kube-api-access-bklvc\" (UniqueName: \"kubernetes.io/projected/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-kube-api-access-bklvc\") pod \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " Oct 31 21:32:59.169296 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.169277 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-xks-ca\") pod \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " Oct 31 21:32:59.169441 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.169424 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-openshift-ca\") pod \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\" (UID: \"35627e3c-c0a8-48a0-9bfd-9732d2385ebd\") " Oct 31 21:32:59.170765 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.170110 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-openshift-ca" pod "35627e3c-c0a8-48a0-9bfd-9732d2385ebd" (UID: "35627e3c-c0a8-48a0-9bfd-9732d2385ebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:32:59.192602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.192555 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-kube-api-access-bklvc" pod "35627e3c-c0a8-48a0-9bfd-9732d2385ebd" (UID: "35627e3c-c0a8-48a0-9bfd-9732d2385ebd"). InnerVolumeSpecName "kube-api-access-bklvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:32:59.218267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.218006 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-xks-ca" pod "35627e3c-c0a8-48a0-9bfd-9732d2385ebd" (UID: "35627e3c-c0a8-48a0-9bfd-9732d2385ebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:32:59.275363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.271607 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bklvc\" (UniqueName: \"kubernetes.io/projected/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-kube-api-access-bklvc\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:59.275363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.271643 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:59.275363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.271657 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35627e3c-c0a8-48a0-9bfd-9732d2385ebd-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:32:59.444590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.444562 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" Oct 31 21:32:59.444590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.444566 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26" event={"ID":"35627e3c-c0a8-48a0-9bfd-9732d2385ebd","Type":"ContainerDied","Data":"f05c5219bb38621c4bb0c4de0e8b512b151bf055f7e4c0f97e772f7fdfcebe87"} Oct 31 21:32:59.444819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.444625 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f05c5219bb38621c4bb0c4de0e8b512b151bf055f7e4c0f97e772f7fdfcebe87" Oct 31 21:32:59.527200 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.527170 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk"] Oct 31 21:32:59.528546 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:32:59.528507 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62385c77_0dc0_43b8_b4d0_a6fd89b2bb45.slice/crio-ed99f8172edc38c08e7ae16b0d7841fb922c7991f43a3434482ecbc8980dc52a WatchSource:0}: Error finding container ed99f8172edc38c08e7ae16b0d7841fb922c7991f43a3434482ecbc8980dc52a: Status 404 returned error can't find the container with id ed99f8172edc38c08e7ae16b0d7841fb922c7991f43a3434482ecbc8980dc52a Oct 31 21:32:59.618643 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.618340 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 31 21:32:59.618643 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:32:59.618431 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 31 21:33:00.156124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:00.156078 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 21:33:00.156354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:00.156169 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 21:33:00.469457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:00.469361 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" event={"ID":"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45","Type":"ContainerStarted","Data":"4019666bb5bbd66d201bfb61e593e89c5ecd9e06b851f934c79caefdc237971f"} Oct 31 21:33:00.469762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:00.469741 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" event={"ID":"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45","Type":"ContainerStarted","Data":"ed99f8172edc38c08e7ae16b0d7841fb922c7991f43a3434482ecbc8980dc52a"} Oct 31 21:33:00.517156 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:00.517090 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" podStartSLOduration=2.517073525 podStartE2EDuration="2.517073525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:00.510956835 +0000 UTC m=+1027.203914227" watchObservedRunningTime="2026-10-31 21:33:00.517073525 +0000 UTC m=+1027.210030917" Oct 31 21:33:01.106427 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:01.106388 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:33:01.106953 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:01.106932 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:33:01.108583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:01.108524 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:33:01.155914 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:01.155878 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 31 21:33:01.156126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:01.155942 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 31 21:33:01.480451 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:01.480415 5188 generic.go:356] "Generic (PLEG): container finished" podID="62385c77-0dc0-43b8-b4d0-a6fd89b2bb45" containerID="4019666bb5bbd66d201bfb61e593e89c5ecd9e06b851f934c79caefdc237971f" exitCode=0 Oct 31 21:33:01.481042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:01.481009 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" event={"ID":"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45","Type":"ContainerDied","Data":"4019666bb5bbd66d201bfb61e593e89c5ecd9e06b851f934c79caefdc237971f"} Oct 31 21:33:02.166936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.166902 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:33:02.167527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.167474 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:33:02.168395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.168372 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:33:02.565498 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:33:02.565456 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:33:02.687362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.687337 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:33:02.748872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.748774 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-openshift-ca\") pod \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " Oct 31 21:33:02.749081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.749047 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-kube-api-access-zvz7n\" (UniqueName: \"kubernetes.io/projected/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-kube-api-access-zvz7n\") pod \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " Oct 31 21:33:02.749171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.749103 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-xks-ca\") pod \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\" (UID: \"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45\") " Oct 31 21:33:02.749247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.749215 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-openshift-ca" pod "62385c77-0dc0-43b8-b4d0-a6fd89b2bb45" (UID: "62385c77-0dc0-43b8-b4d0-a6fd89b2bb45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:33:02.749822 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.749795 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:33:02.751371 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.751338 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-xks-ca" pod "62385c77-0dc0-43b8-b4d0-a6fd89b2bb45" (UID: "62385c77-0dc0-43b8-b4d0-a6fd89b2bb45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:33:02.751712 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.751684 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-kube-api-access-zvz7n" pod "62385c77-0dc0-43b8-b4d0-a6fd89b2bb45" (UID: "62385c77-0dc0-43b8-b4d0-a6fd89b2bb45"). InnerVolumeSpecName "kube-api-access-zvz7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:33:02.851651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.851611 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvz7n\" (UniqueName: \"kubernetes.io/projected/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-kube-api-access-zvz7n\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:33:02.851651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:02.851646 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:33:03.503430 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:03.503378 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" event={"ID":"62385c77-0dc0-43b8-b4d0-a6fd89b2bb45","Type":"ContainerDied","Data":"ed99f8172edc38c08e7ae16b0d7841fb922c7991f43a3434482ecbc8980dc52a"} Oct 31 21:33:03.503850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:03.503466 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed99f8172edc38c08e7ae16b0d7841fb922c7991f43a3434482ecbc8980dc52a" Oct 31 21:33:03.503850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:03.503403 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk" Oct 31 21:33:07.289918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.289825 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx"] Oct 31 21:33:07.290798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.290777 5188 cpu_manager.go:414] "containerMap: removing container" podUID="62385c77-0dc0-43b8-b4d0-a6fd89b2bb45" containerName="revoke-keys" Oct 31 21:33:07.290900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.290799 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62385c77-0dc0-43b8-b4d0-a6fd89b2bb45" containerName="revoke-keys" Oct 31 21:33:07.290900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.290858 5188 cpu_manager.go:414] "containerMap: removing container" podUID="35627e3c-c0a8-48a0-9bfd-9732d2385ebd" containerName="revoke-keys" Oct 31 21:33:07.290900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.290866 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35627e3c-c0a8-48a0-9bfd-9732d2385ebd" containerName="revoke-keys" Oct 31 21:33:07.291058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.291012 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="62385c77-0dc0-43b8-b4d0-a6fd89b2bb45" containerName="revoke-keys" Oct 31 21:33:07.291058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.291034 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="35627e3c-c0a8-48a0-9bfd-9732d2385ebd" containerName="revoke-keys" Oct 31 21:33:07.296046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.296024 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.323875 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.323841 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:33:07.324226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.324198 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:33:07.324572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.324549 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:33:07.324926 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.324904 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:33:07.325058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.324904 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx"] Oct 31 21:33:07.412666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.412626 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82gbj\" (UniqueName: \"kubernetes.io/projected/ba70b88d-77ab-49f2-b66b-2b005d8962ac-kube-api-access-82gbj\") pod \"maas-api-revoke-sub-5874c9d2-tgrgx\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.412929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.412910 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5874c9d2-tgrgx\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.412992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.412966 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5874c9d2-tgrgx\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.514161 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.514112 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5874c9d2-tgrgx\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.514390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.514165 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82gbj\" (UniqueName: \"kubernetes.io/projected/ba70b88d-77ab-49f2-b66b-2b005d8962ac-kube-api-access-82gbj\") pod \"maas-api-revoke-sub-5874c9d2-tgrgx\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.514474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.514453 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5874c9d2-tgrgx\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.515162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.515128 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5874c9d2-tgrgx\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.516726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.516695 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5874c9d2-tgrgx\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.535758 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.535732 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82gbj\" (UniqueName: \"kubernetes.io/projected/ba70b88d-77ab-49f2-b66b-2b005d8962ac-kube-api-access-82gbj\") pod \"maas-api-revoke-sub-5874c9d2-tgrgx\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.611718 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.611369 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:07.958975 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:33:07.958937 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba70b88d_77ab_49f2_b66b_2b005d8962ac.slice/crio-2a5a8cdcfd73b26f2073f315094046728e61fa23686e7babd16a695bf3cbe9ad WatchSource:0}: Error finding container 2a5a8cdcfd73b26f2073f315094046728e61fa23686e7babd16a695bf3cbe9ad: Status 404 returned error can't find the container with id 2a5a8cdcfd73b26f2073f315094046728e61fa23686e7babd16a695bf3cbe9ad Oct 31 21:33:07.960017 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:07.959994 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx"] Oct 31 21:33:08.551285 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:08.551197 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" event={"ID":"ba70b88d-77ab-49f2-b66b-2b005d8962ac","Type":"ContainerStarted","Data":"50f6ca844c8cf38fa884365f3a40bb594326c1379efec5d18bcb6156cd1e92e9"} Oct 31 21:33:08.551285 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:08.551251 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" event={"ID":"ba70b88d-77ab-49f2-b66b-2b005d8962ac","Type":"ContainerStarted","Data":"2a5a8cdcfd73b26f2073f315094046728e61fa23686e7babd16a695bf3cbe9ad"} Oct 31 21:33:08.589642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:08.589592 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" podStartSLOduration=1.589578677 podStartE2EDuration="1.589578677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:08.589225527 +0000 UTC m=+1035.282182920" watchObservedRunningTime="2026-10-31 21:33:08.589578677 +0000 UTC m=+1035.282536070" Oct 31 21:33:09.664348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:09.658385 5188 generic.go:356] "Generic (PLEG): container finished" podID="ba70b88d-77ab-49f2-b66b-2b005d8962ac" containerID="50f6ca844c8cf38fa884365f3a40bb594326c1379efec5d18bcb6156cd1e92e9" exitCode=0 Oct 31 21:33:09.664348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:09.658833 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" event={"ID":"ba70b88d-77ab-49f2-b66b-2b005d8962ac","Type":"ContainerDied","Data":"50f6ca844c8cf38fa884365f3a40bb594326c1379efec5d18bcb6156cd1e92e9"} Oct 31 21:33:10.573348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:10.572802 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 21:33:10.597354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:10.596473 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:10.843422 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:10.829426 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Oct 31 21:33:10.935357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:10.901565 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz"] Oct 31 21:33:10.946394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:10.941444 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:10.977970 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:10.977911 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:33:11.004245 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:10.999727 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.020927 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.020869 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz"] Oct 31 21:33:11.157349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.153707 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-90db52-openshift-default-dockercfg-7wqrw\"" Oct 31 21:33:11.445406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.417774 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-envoy\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.445406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.417904 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-podinfo\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.445406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.418029 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-credential-socket\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.445406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.418077 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-data\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.445406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.418116 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-socket\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.445406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.418147 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ww2s\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-kube-api-access-7ww2s\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.445406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.418262 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378f9a55-a031-406d-bffc-ccc689ac4f29-istiod-ca-cert\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.445406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.418395 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-token\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.445406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.418463 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-certs\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.559304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.559262 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378f9a55-a031-406d-bffc-ccc689ac4f29-istiod-ca-cert\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.559658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.559635 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-token\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.559840 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.559820 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-certs\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.560053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.560035 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-envoy\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.560233 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.560215 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-podinfo\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.560493 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.560470 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-credential-socket\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.560646 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.560630 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-data\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.560781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.560766 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-socket\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.560903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.560886 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ww2s\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-kube-api-access-7ww2s\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.586139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.586092 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378f9a55-a031-406d-bffc-ccc689ac4f29-istiod-ca-cert\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.606230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.586452 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.609392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.609361 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-certs\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.858593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.610746 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-credential-socket\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.858593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.610983 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-data\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.858593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.611196 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-socket\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.858593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.631012 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-podinfo\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.858593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.639970 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.885229 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.669434 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-token\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:11.955074 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:11.955015 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.153358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:12.131795 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:33:12.372594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:12.372560 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:33:12.372906 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:12.372888 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:33:12.393350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:12.197510 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-envoy\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:12.394442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:12.200683 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ww2s\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-kube-api-access-7ww2s\") pod \"e2e-api-a-90db52-openshift-default-f788579b9-xp2jz\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:12.394442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:12.201387 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.394965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:12.394936 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:33:12.622460 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:12.610438 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.828036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:12.827989 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:13.080625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:13.078821 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:13.080625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:13.079064 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 31 21:33:13.372812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:13.372764 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 31 21:33:13.763368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:13.394405 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:13.920666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:13.920608 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:14.326406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:14.272741 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:33:14.413401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:14.272789 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Oct 31 21:33:14.472625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:14.287939 5188 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:14.629420 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:33:14.606076 5188 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.115s" Oct 31 21:33:14.629780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:14.629752 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:33:15.102031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.100846 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-cts4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:33:15.102031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.100921 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:15.252851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.252822 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz"] Oct 31 21:33:15.271971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.256935 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:15.272748 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:33:15.272711 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378f9a55_a031_406d_bffc_ccc689ac4f29.slice/crio-9ad8f81359742fa8a3db74379097ac3023921cbb710401545731baa2e47c47ae WatchSource:0}: Error finding container 9ad8f81359742fa8a3db74379097ac3023921cbb710401545731baa2e47c47ae: Status 404 returned error can't find the container with id 9ad8f81359742fa8a3db74379097ac3023921cbb710401545731baa2e47c47ae Oct 31 21:33:15.282526 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.282477 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:15.282827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.282800 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:15.282984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.282964 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:15.318474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.295773 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-xks-ca\") pod \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " Oct 31 21:33:15.318474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.295883 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-openshift-ca\") pod \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " Oct 31 21:33:15.318474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.296074 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba70b88d-77ab-49f2-b66b-2b005d8962ac-kube-api-access-82gbj\" (UniqueName: \"kubernetes.io/projected/ba70b88d-77ab-49f2-b66b-2b005d8962ac-kube-api-access-82gbj\") pod \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\" (UID: \"ba70b88d-77ab-49f2-b66b-2b005d8962ac\") " Oct 31 21:33:15.318474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.316020 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-openshift-ca" pod "ba70b88d-77ab-49f2-b66b-2b005d8962ac" (UID: "ba70b88d-77ab-49f2-b66b-2b005d8962ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:33:15.320386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.320349 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 21:33:15.350516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.338991 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:33:15.423344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.410213 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:33:15.583391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.440618 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:15.583391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.443879 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 21:33:15.583391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.538122 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:33:15.583391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.538177 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:15.583391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.538902 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba70b88d-77ab-49f2-b66b-2b005d8962ac-kube-api-access-82gbj" pod "ba70b88d-77ab-49f2-b66b-2b005d8962ac" (UID: "ba70b88d-77ab-49f2-b66b-2b005d8962ac"). InnerVolumeSpecName "kube-api-access-82gbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:33:15.588585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.587469 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-xks-ca" pod "ba70b88d-77ab-49f2-b66b-2b005d8962ac" (UID: "ba70b88d-77ab-49f2-b66b-2b005d8962ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:33:15.653348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.638256 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82gbj\" (UniqueName: \"kubernetes.io/projected/ba70b88d-77ab-49f2-b66b-2b005d8962ac-kube-api-access-82gbj\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:33:15.653348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:15.638303 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba70b88d-77ab-49f2-b66b-2b005d8962ac-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:33:16.349500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.349458 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh"] Oct 31 21:33:16.353524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.353489 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ba70b88d-77ab-49f2-b66b-2b005d8962ac" containerName="revoke-keys" Oct 31 21:33:16.354095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.354072 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba70b88d-77ab-49f2-b66b-2b005d8962ac" containerName="revoke-keys" Oct 31 21:33:16.355341 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.355303 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba70b88d-77ab-49f2-b66b-2b005d8962ac" containerName="revoke-keys" Oct 31 21:33:16.371028 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.370995 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.436362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.412224 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:33:16.457241 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.421159 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-084d807f-openshift-default-dockercfg-8hmhd\"" Oct 31 21:33:16.492772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.492403 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh"] Oct 31 21:33:16.635948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.610894 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-certs\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.635948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.611020 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-envoy\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.635948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.611086 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istiod-ca-cert\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.635948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.611224 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvm4\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-kube-api-access-hxvm4\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.635948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.611295 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.635948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.611334 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-podinfo\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.635948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.611370 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-data\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.635948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.611420 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-token\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.635948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.611495 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-credential-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.670639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.660917 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" event={"ID":"ba70b88d-77ab-49f2-b66b-2b005d8962ac","Type":"ContainerDied","Data":"2a5a8cdcfd73b26f2073f315094046728e61fa23686e7babd16a695bf3cbe9ad"} Oct 31 21:33:16.670639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.660986 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a5a8cdcfd73b26f2073f315094046728e61fa23686e7babd16a695bf3cbe9ad" Oct 31 21:33:16.670639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.661168 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx" Oct 31 21:33:16.671797 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.671744 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" event={"ID":"378f9a55-a031-406d-bffc-ccc689ac4f29","Type":"ContainerStarted","Data":"9ad8f81359742fa8a3db74379097ac3023921cbb710401545731baa2e47c47ae"} Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720069 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-certs\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720142 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-envoy\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720166 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istiod-ca-cert\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720227 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvm4\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-kube-api-access-hxvm4\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720259 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720276 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-podinfo\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720298 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-data\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720347 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-token\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720399 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-credential-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.720937 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-credential-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.728827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.721175 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-certs\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.741920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.729557 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-envoy\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.741920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.730380 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istiod-ca-cert\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.741920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.739360 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.746044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.745947 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" podStartSLOduration=6.745924119 podStartE2EDuration="6.745924119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:16.740238434 +0000 UTC m=+1043.433195826" watchObservedRunningTime="2026-10-31 21:33:16.745924119 +0000 UTC m=+1043.438881514" Oct 31 21:33:16.749551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.749511 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-podinfo\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.749904 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.749880 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-data\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.772108 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.772077 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvm4\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-kube-api-access-hxvm4\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.772333 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.772183 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-token\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-zqhjh\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:16.856383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:16.853024 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:17.087261 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.087171 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:17.088544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.088512 5188 patch_prober.go:29] interesting pod/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 21:33:17.088824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.088787 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 21:33:17.225815 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:33:17.225772 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96111e4f_31bc_4769_a00a_ddf36fa4e2af.slice/crio-fa3249b239d977625eece916fc0e5037f9b003c20dd2b51daadb079eef72da9c WatchSource:0}: Error finding container fa3249b239d977625eece916fc0e5037f9b003c20dd2b51daadb079eef72da9c: Status 404 returned error can't find the container with id fa3249b239d977625eece916fc0e5037f9b003c20dd2b51daadb079eef72da9c Oct 31 21:33:17.226450 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.226424 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh"] Oct 31 21:33:17.228463 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.228429 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:17.228559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.228519 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:17.228627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.228569 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:17.646959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.646918 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww"] Oct 31 21:33:17.652544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.652518 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.669818 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.669788 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww"] Oct 31 21:33:17.673834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.673790 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:33:17.673985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.673937 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:33:17.674068 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.673996 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:33:17.674068 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.674020 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:33:17.685037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.685002 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" event={"ID":"96111e4f-31bc-4769-a00a-ddf36fa4e2af","Type":"ContainerStarted","Data":"a6f0d37a791933965eb010150c0988434006bc639ee6ea88a596c3cd10bced1c"} Oct 31 21:33:17.685235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.685068 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" event={"ID":"96111e4f-31bc-4769-a00a-ddf36fa4e2af","Type":"ContainerStarted","Data":"fa3249b239d977625eece916fc0e5037f9b003c20dd2b51daadb079eef72da9c"} Oct 31 21:33:17.687577 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.687498 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" event={"ID":"378f9a55-a031-406d-bffc-ccc689ac4f29","Type":"ContainerStarted","Data":"1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738"} Oct 31 21:33:17.734989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.734931 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" podStartSLOduration=2.7349179980000002 podStartE2EDuration="2.734917998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:17.731225059 +0000 UTC m=+1044.424182466" watchObservedRunningTime="2026-10-31 21:33:17.734917998 +0000 UTC m=+1044.427875390" Oct 31 21:33:17.744497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.744467 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1870b67-bkzww\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.744676 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.744535 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1870b67-bkzww\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.744726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.744680 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgh2\" (UniqueName: \"kubernetes.io/projected/9883fcca-9fdf-438f-9688-a435b5154ce4-kube-api-access-5jgh2\") pod \"maas-api-revoke-sub-a1870b67-bkzww\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.846741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.846695 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgh2\" (UniqueName: \"kubernetes.io/projected/9883fcca-9fdf-438f-9688-a435b5154ce4-kube-api-access-5jgh2\") pod \"maas-api-revoke-sub-a1870b67-bkzww\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.846973 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.846949 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1870b67-bkzww\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.847072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.847056 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1870b67-bkzww\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.847835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.847811 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1870b67-bkzww\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.850357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.849702 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1870b67-bkzww\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.876406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.863049 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:17.876406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.876141 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgh2\" (UniqueName: \"kubernetes.io/projected/9883fcca-9fdf-438f-9688-a435b5154ce4-kube-api-access-5jgh2\") pod \"maas-api-revoke-sub-a1870b67-bkzww\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:17.970706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:17.970566 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:18.092427 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:18.092390 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:18.280755 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:33:18.280709 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9883fcca_9fdf_438f_9688_a435b5154ce4.slice/crio-a7b08a9ee6cbb275aebe2933aa77a9be3b1564bc40c93e95c0b6dcd3bf4ff46e WatchSource:0}: Error finding container a7b08a9ee6cbb275aebe2933aa77a9be3b1564bc40c93e95c0b6dcd3bf4ff46e: Status 404 returned error can't find the container with id a7b08a9ee6cbb275aebe2933aa77a9be3b1564bc40c93e95c0b6dcd3bf4ff46e Oct 31 21:33:18.281654 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:18.281628 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww"] Oct 31 21:33:18.698198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:18.698152 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" event={"ID":"9883fcca-9fdf-438f-9688-a435b5154ce4","Type":"ContainerStarted","Data":"e389c51ab92e1e8cab261d60fe53ad21bf1c4387a549a3680a9683febcd78eaa"} Oct 31 21:33:18.698697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:18.698219 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" event={"ID":"9883fcca-9fdf-438f-9688-a435b5154ce4","Type":"ContainerStarted","Data":"a7b08a9ee6cbb275aebe2933aa77a9be3b1564bc40c93e95c0b6dcd3bf4ff46e"} Oct 31 21:33:18.698697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:18.698600 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:18.699765 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:18.699742 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:33:18.737275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:18.737211 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" podStartSLOduration=1.737191664 podStartE2EDuration="1.737191664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:18.733684189 +0000 UTC m=+1045.426641580" watchObservedRunningTime="2026-10-31 21:33:18.737191664 +0000 UTC m=+1045.430149056" Oct 31 21:33:18.865234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:18.865202 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:19.710406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:19.710365 5188 generic.go:356] "Generic (PLEG): container finished" podID="9883fcca-9fdf-438f-9688-a435b5154ce4" containerID="e389c51ab92e1e8cab261d60fe53ad21bf1c4387a549a3680a9683febcd78eaa" exitCode=0 Oct 31 21:33:19.710839 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:19.710466 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" event={"ID":"9883fcca-9fdf-438f-9688-a435b5154ce4","Type":"ContainerDied","Data":"e389c51ab92e1e8cab261d60fe53ad21bf1c4387a549a3680a9683febcd78eaa"} Oct 31 21:33:19.711079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:19.711062 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:19.712010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:19.711986 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:33:20.944572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:20.944524 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:21.110647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.110610 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9883fcca-9fdf-438f-9688-a435b5154ce4-kube-api-access-5jgh2\" (UniqueName: \"kubernetes.io/projected/9883fcca-9fdf-438f-9688-a435b5154ce4-kube-api-access-5jgh2\") pod \"9883fcca-9fdf-438f-9688-a435b5154ce4\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " Oct 31 21:33:21.110818 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.110706 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-openshift-ca\") pod \"9883fcca-9fdf-438f-9688-a435b5154ce4\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " Oct 31 21:33:21.110877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.110854 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-xks-ca\") pod \"9883fcca-9fdf-438f-9688-a435b5154ce4\" (UID: \"9883fcca-9fdf-438f-9688-a435b5154ce4\") " Oct 31 21:33:21.111156 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.111122 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-openshift-ca" pod "9883fcca-9fdf-438f-9688-a435b5154ce4" (UID: "9883fcca-9fdf-438f-9688-a435b5154ce4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:33:21.111660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.111635 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:33:21.112963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.112934 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-xks-ca" pod "9883fcca-9fdf-438f-9688-a435b5154ce4" (UID: "9883fcca-9fdf-438f-9688-a435b5154ce4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:33:21.113052 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.112960 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9883fcca-9fdf-438f-9688-a435b5154ce4-kube-api-access-5jgh2" pod "9883fcca-9fdf-438f-9688-a435b5154ce4" (UID: "9883fcca-9fdf-438f-9688-a435b5154ce4"). InnerVolumeSpecName "kube-api-access-5jgh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:33:21.256949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.256873 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgh2\" (UniqueName: \"kubernetes.io/projected/9883fcca-9fdf-438f-9688-a435b5154ce4-kube-api-access-5jgh2\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:33:21.256949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.256915 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9883fcca-9fdf-438f-9688-a435b5154ce4-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:33:21.731896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.731863 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" Oct 31 21:33:21.732124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.731897 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww" event={"ID":"9883fcca-9fdf-438f-9688-a435b5154ce4","Type":"ContainerDied","Data":"a7b08a9ee6cbb275aebe2933aa77a9be3b1564bc40c93e95c0b6dcd3bf4ff46e"} Oct 31 21:33:21.732124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:21.731955 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7b08a9ee6cbb275aebe2933aa77a9be3b1564bc40c93e95c0b6dcd3bf4ff46e" Oct 31 21:33:30.127349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.126809 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf"] Oct 31 21:33:30.245535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.127921 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9883fcca-9fdf-438f-9688-a435b5154ce4" containerName="revoke-keys" Oct 31 21:33:30.245535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.127944 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9883fcca-9fdf-438f-9688-a435b5154ce4" containerName="revoke-keys" Oct 31 21:33:30.245535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.128156 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9883fcca-9fdf-438f-9688-a435b5154ce4" containerName="revoke-keys" Oct 31 21:33:30.245535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.195470 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.245535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.219297 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7ac3c8-openshift-default-dockercfg-l27zs\"" Oct 31 21:33:30.291452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.290371 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf"] Oct 31 21:33:30.373538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.362258 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istiod-ca-cert\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.373917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.373881 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-certs\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.374159 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.374136 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-token\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.374296 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.374279 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-podinfo\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.374552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.374528 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-credential-socket\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.374752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.374734 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-socket\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.374887 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.374871 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcxlz\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-kube-api-access-hcxlz\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.375049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.375031 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-envoy\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.375190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.375174 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-data\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.496585 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istiod-ca-cert\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.496681 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-certs\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.496775 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-token\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.496806 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-podinfo\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.496886 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-credential-socket\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.496965 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-socket\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.496999 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcxlz\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-kube-api-access-hcxlz\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.497057 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-envoy\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.497095 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-data\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.497734 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-data\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.498375 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istiod-ca-cert\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.507683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.498616 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-certs\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.514279 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.514230 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-socket\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.515587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.514801 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-credential-socket\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.526358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.523144 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-token\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.543598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.543494 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-envoy\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.544527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.544502 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-podinfo\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.565168 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.565125 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcxlz\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-kube-api-access-hcxlz\") pod \"e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:30.651335 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:30.651283 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:31.085853 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:33:31.085816 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae718dc5_7881_4f3f_a98d_a517cefd0ab4.slice/crio-c29ee9d0177e1ae1d29f5c83177fd118b310b7765145826f0ac88840aacd4b15 WatchSource:0}: Error finding container c29ee9d0177e1ae1d29f5c83177fd118b310b7765145826f0ac88840aacd4b15: Status 404 returned error can't find the container with id c29ee9d0177e1ae1d29f5c83177fd118b310b7765145826f0ac88840aacd4b15 Oct 31 21:33:31.087299 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:31.087263 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf"] Oct 31 21:33:31.088524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:31.088492 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:31.088597 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:31.088561 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:31.088597 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:31.088593 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:31.955554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:31.955252 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" event={"ID":"ae718dc5-7881-4f3f-a98d-a517cefd0ab4","Type":"ContainerStarted","Data":"827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d"} Oct 31 21:33:31.955554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:31.955447 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" event={"ID":"ae718dc5-7881-4f3f-a98d-a517cefd0ab4","Type":"ContainerStarted","Data":"c29ee9d0177e1ae1d29f5c83177fd118b310b7765145826f0ac88840aacd4b15"} Oct 31 21:33:32.016186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:32.016123 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" podStartSLOduration=3.016105198 podStartE2EDuration="3.016105198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:32.015655995 +0000 UTC m=+1058.708613391" watchObservedRunningTime="2026-10-31 21:33:32.016105198 +0000 UTC m=+1058.709062591" Oct 31 21:33:32.538192 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:33:32.538154 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:33:32.678884 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:32.678847 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:32.685295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:32.685263 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:32.965037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:32.964996 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:32.966043 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:32.966023 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:33:36.924493 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:36.924452 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8"] Oct 31 21:33:36.929532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:36.929501 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:36.952757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:36.952723 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7ac3c8-dockercfg-mxj8v\"" Oct 31 21:33:36.952757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:36.952756 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7ac3c8\"" Oct 31 21:33:36.960376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:36.960315 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8"] Oct 31 21:33:37.263371 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:37.263241 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnnlr\" (UniqueName: \"kubernetes.io/projected/8b0f6229-c18a-46ee-b93b-5475367bee46-kube-api-access-mnnlr\") pod \"payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8\" (UID: \"8b0f6229-c18a-46ee-b93b-5475367bee46\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:37.263552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:37.263444 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b0f6229-c18a-46ee-b93b-5475367bee46-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8\" (UID: \"8b0f6229-c18a-46ee-b93b-5475367bee46\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:37.364750 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:37.364707 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnnlr\" (UniqueName: \"kubernetes.io/projected/8b0f6229-c18a-46ee-b93b-5475367bee46-kube-api-access-mnnlr\") pod \"payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8\" (UID: \"8b0f6229-c18a-46ee-b93b-5475367bee46\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:37.364958 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:37.364818 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b0f6229-c18a-46ee-b93b-5475367bee46-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8\" (UID: \"8b0f6229-c18a-46ee-b93b-5475367bee46\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:37.365587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:37.365564 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b0f6229-c18a-46ee-b93b-5475367bee46-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8\" (UID: \"8b0f6229-c18a-46ee-b93b-5475367bee46\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:37.392530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:37.392498 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnnlr\" (UniqueName: \"kubernetes.io/projected/8b0f6229-c18a-46ee-b93b-5475367bee46-kube-api-access-mnnlr\") pod \"payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8\" (UID: \"8b0f6229-c18a-46ee-b93b-5475367bee46\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:37.547262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:37.547170 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:37.923935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:37.923900 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8"] Oct 31 21:33:37.924667 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:33:37.924635 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0f6229_c18a_46ee_b93b_5475367bee46.slice/crio-243dd5aa93f01e9a5738d30acaba50249cd4ea33f94e5f3850bf1c50c0247f7c WatchSource:0}: Error finding container 243dd5aa93f01e9a5738d30acaba50249cd4ea33f94e5f3850bf1c50c0247f7c: Status 404 returned error can't find the container with id 243dd5aa93f01e9a5738d30acaba50249cd4ea33f94e5f3850bf1c50c0247f7c Oct 31 21:33:38.169915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:38.169866 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" event={"ID":"8b0f6229-c18a-46ee-b93b-5475367bee46","Type":"ContainerStarted","Data":"84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b"} Oct 31 21:33:38.170115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:38.169936 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" event={"ID":"8b0f6229-c18a-46ee-b93b-5475367bee46","Type":"ContainerStarted","Data":"243dd5aa93f01e9a5738d30acaba50249cd4ea33f94e5f3850bf1c50c0247f7c"} Oct 31 21:33:38.170115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:38.169969 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:38.210097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:38.209982 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" podStartSLOduration=2.209959699 podStartE2EDuration="2.209959699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:38.205674836 +0000 UTC m=+1064.898632229" watchObservedRunningTime="2026-10-31 21:33:38.209959699 +0000 UTC m=+1064.902917092" Oct 31 21:33:38.935074 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:38.935035 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx"] Oct 31 21:33:38.940417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:38.940390 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:38.973272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:38.973221 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx"] Oct 31 21:33:39.038559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.038522 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.038559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.038570 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd42e3ee-f036-4bc4-87ab-45103198c952-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.038842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.038633 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.038842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.038656 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.038842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.038719 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.038842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.038778 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.038842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.038818 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2mkf\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-kube-api-access-f2mkf\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.041410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.038877 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.041410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.038992 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.140453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140393 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.140453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140435 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.140731 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140474 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.140731 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140511 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.140731 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140539 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2mkf\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-kube-api-access-f2mkf\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.140731 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140567 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.140731 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140618 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.141291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140874 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.141291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140926 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd42e3ee-f036-4bc4-87ab-45103198c952-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.141291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140930 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.141291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.140990 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.141291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.141148 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.141291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.141214 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.141793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.141771 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd42e3ee-f036-4bc4-87ab-45103198c952-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.142947 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.142924 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.143085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.143068 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.163035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.163002 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.163466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.163435 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2mkf\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-kube-api-access-f2mkf\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.257658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.257555 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:39.642314 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:33:39.642271 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd42e3ee_f036_4bc4_87ab_45103198c952.slice/crio-c48ddd60905c93b623c69333f59cfb94e1cb773702d7ff387ebefe205fd49788 WatchSource:0}: Error finding container c48ddd60905c93b623c69333f59cfb94e1cb773702d7ff387ebefe205fd49788: Status 404 returned error can't find the container with id c48ddd60905c93b623c69333f59cfb94e1cb773702d7ff387ebefe205fd49788 Oct 31 21:33:39.643661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.643637 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx"] Oct 31 21:33:39.645406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.645371 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:39.645513 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.645443 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:39.645513 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:39.645496 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:40.194772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:40.194732 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" event={"ID":"bd42e3ee-f036-4bc4-87ab-45103198c952","Type":"ContainerStarted","Data":"431a263cd0f1114c730a30f63e5a0a6e3e1ae2f1b2a4e48fa10f81df3284a884"} Oct 31 21:33:40.195301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:40.194791 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" event={"ID":"bd42e3ee-f036-4bc4-87ab-45103198c952","Type":"ContainerStarted","Data":"c48ddd60905c93b623c69333f59cfb94e1cb773702d7ff387ebefe205fd49788"} Oct 31 21:33:40.257749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:40.257717 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:40.258891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:40.258841 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podStartSLOduration=2.258826914 podStartE2EDuration="2.258826914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:40.244505182 +0000 UTC m=+1066.937462576" watchObservedRunningTime="2026-10-31 21:33:40.258826914 +0000 UTC m=+1066.951784306" Oct 31 21:33:40.259486 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:40.259460 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 21:33:40.259608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:40.259545 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 21:33:41.258223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:41.258192 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 21:33:41.258750 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:41.258722 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 21:33:42.258809 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:42.258701 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 21:33:42.258809 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:42.258756 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 21:33:43.258213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:43.258155 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 21:33:43.258551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:43.258239 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 21:33:44.260761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:44.260720 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 21:33:44.261373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:44.261337 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 21:33:45.258838 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:45.258792 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 21:33:45.259193 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:45.259132 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 21:33:46.120578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:45.924619 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9"] Oct 31 21:33:46.129492 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.129455 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9"] Oct 31 21:33:46.129878 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.129785 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.169315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.169283 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wxtz5\"" Oct 31 21:33:46.170230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.169614 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 21:33:46.170491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.169653 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 21:33:46.170613 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.169696 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 21:33:46.258800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.258372 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f930b93-d00a-4c4f-a3be-bc32841ca19e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.259162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.259136 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.259510 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.259489 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.259742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.259719 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.259944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.259926 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.261442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.261418 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pjxk\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-kube-api-access-2pjxk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.261713 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.261692 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.270296 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.269720 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 21:33:46.270296 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.269785 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 21:33:46.368835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.366413 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.368835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.366512 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.368835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.366553 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjxk\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-kube-api-access-2pjxk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.368835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.366638 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.368835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.366733 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f930b93-d00a-4c4f-a3be-bc32841ca19e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.368835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.366809 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.368835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.366978 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.370316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.370282 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.428438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.423066 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f930b93-d00a-4c4f-a3be-bc32841ca19e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.429112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.429082 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.451678 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.441459 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.452040 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.449196 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.452842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.452798 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjxk\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-kube-api-access-2pjxk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.455995 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.455949 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r4hv9\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:46.518726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:46.518685 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:47.258926 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:47.258891 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 31 21:33:47.259355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:47.258946 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 31 21:33:47.340085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:47.340047 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9"] Oct 31 21:33:47.342956 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:47.342928 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:33:47.523924 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:47.523827 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" event={"ID":"0f930b93-d00a-4c4f-a3be-bc32841ca19e","Type":"ContainerStarted","Data":"7579a503bacea5ad3f7908cf36ef0b602e117ea6a1dcecef094b03e9cde3f6f3"} Oct 31 21:33:48.762999 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:48.762949 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:48.765051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:48.765030 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:48.766495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:48.766432 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 21:33:49.230734 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:49.230691 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:33:49.827380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:49.826135 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4"] Oct 31 21:33:49.840716 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:49.840680 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:49.848111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:49.848054 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4"] Oct 31 21:33:50.053532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.053486 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.053807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.053784 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.054076 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.054055 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.054194 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.054180 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.054359 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.054345 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.054554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.054517 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.054732 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.054708 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpxv5\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-kube-api-access-kpxv5\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.054807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.054793 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.054951 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.054926 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.156203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.156166 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.156433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.156236 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.156433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.156276 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.156433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.156299 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.156433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.156415 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.156661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.156442 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.156661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.156513 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.156661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.156541 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.156816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.156667 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpxv5\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-kube-api-access-kpxv5\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.157074 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.157048 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.157431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.157401 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.157709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.157688 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.157791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.157687 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.157989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.157969 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.160795 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.160772 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.166908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.166854 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.218252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.218175 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpxv5\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-kube-api-access-kpxv5\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.226349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.224928 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:50.249476 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:50.249179 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:51.341351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:51.341298 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4"] Oct 31 21:33:51.344842 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:33:51.344786 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b0e627_0cf3_49cc_b08a_136f07cc07fb.slice/crio-c507647c75f33cf98a4e12b7acd8a747fda53e4c7aed48ccd4fd040665d5be66 WatchSource:0}: Error finding container c507647c75f33cf98a4e12b7acd8a747fda53e4c7aed48ccd4fd040665d5be66: Status 404 returned error can't find the container with id c507647c75f33cf98a4e12b7acd8a747fda53e4c7aed48ccd4fd040665d5be66 Oct 31 21:33:51.349173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:51.349122 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:51.349300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:51.349243 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:51.349358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:51.349301 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:51.968001 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:51.967134 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" event={"ID":"67b0e627-0cf3-49cc-b08a-136f07cc07fb","Type":"ContainerStarted","Data":"64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871"} Oct 31 21:33:51.968001 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:51.967207 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" event={"ID":"67b0e627-0cf3-49cc-b08a-136f07cc07fb","Type":"ContainerStarted","Data":"c507647c75f33cf98a4e12b7acd8a747fda53e4c7aed48ccd4fd040665d5be66"} Oct 31 21:33:52.043895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:52.040238 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podStartSLOduration=3.040218767 podStartE2EDuration="3.040218767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:52.028788918 +0000 UTC m=+1078.721746310" watchObservedRunningTime="2026-10-31 21:33:52.040218767 +0000 UTC m=+1078.733176161" Oct 31 21:33:52.252765 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:52.252630 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:33:52.254019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:52.253685 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 21:33:52.254019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:52.253743 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 21:33:53.043417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:53.043378 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb"] Oct 31 21:33:53.055188 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:53.055138 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5605c0d-jfnjb"] Oct 31 21:33:53.250794 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:53.250754 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 21:33:53.251005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:53.250823 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 21:33:53.526393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:53.526007 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86db3058-5a9c-4455-89b6-ff9248670691" path="/var/lib/kubelet/pods/86db3058-5a9c-4455-89b6-ff9248670691/volumes" Oct 31 21:33:54.048096 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:54.048011 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh"] Oct 31 21:33:54.085369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:54.070086 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be48c40a-xpbwh"] Oct 31 21:33:54.295358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:54.282541 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 21:33:54.295358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:54.282609 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 21:33:55.252873 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:55.252821 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 21:33:55.253281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:55.252895 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 21:33:55.518390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:55.518256 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4352ee1-e585-4981-805c-b58d7215f6d2" path="/var/lib/kubelet/pods/e4352ee1-e585-4981-805c-b58d7215f6d2/volumes" Oct 31 21:33:56.250165 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:56.250119 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 21:33:56.250429 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:56.250195 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 21:33:56.586943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:56.577484 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:56.586943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:56.577616 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:33:57.131376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:57.130624 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" event={"ID":"0f930b93-d00a-4c4f-a3be-bc32841ca19e","Type":"ContainerStarted","Data":"d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a"} Oct 31 21:33:57.131928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:57.131861 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:57.133844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:57.133653 5188 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r4hv9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:33:57.133844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:57.133710 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" podUID="0f930b93-d00a-4c4f-a3be-bc32841ca19e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:33:57.180344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:57.174869 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" podStartSLOduration=2.945000828 podStartE2EDuration="12.174848106s" totalImagesPullingTime="9.229847278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:57.171502431 +0000 UTC m=+1083.864459827" watchObservedRunningTime="2026-10-31 21:33:57.174848106 +0000 UTC m=+1083.867805500" Oct 31 21:33:57.250280 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:57.250242 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 21:33:57.250998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:57.250615 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 21:33:58.145439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:58.145241 5188 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r4hv9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:33:58.145439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:58.145297 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" podUID="0f930b93-d00a-4c4f-a3be-bc32841ca19e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:33:58.256384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:58.252527 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 21:33:58.256384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:58.252613 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 21:33:59.157040 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:59.156655 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 21:33:59.252627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:59.250703 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 21:33:59.252627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:33:59.250768 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 21:34:00.168804 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.168764 5188 generic.go:356] "Generic (PLEG): container finished" podID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerID="1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738" exitCode=0 Oct 31 21:34:00.169297 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.168825 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" event={"ID":"378f9a55-a031-406d-bffc-ccc689ac4f29","Type":"ContainerDied","Data":"1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738"} Oct 31 21:34:00.169390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.169366 5188 scope.go:122] "RemoveContainer" containerID="1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738" Oct 31 21:34:00.171468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.171429 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:34:00.171602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.171534 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:34:00.171602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.171587 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:34:00.171697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.171625 5188 generic.go:356] "Generic (PLEG): container finished" podID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerID="a6f0d37a791933965eb010150c0988434006bc639ee6ea88a596c3cd10bced1c" exitCode=0 Oct 31 21:34:00.171697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.171664 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" event={"ID":"96111e4f-31bc-4769-a00a-ddf36fa4e2af","Type":"ContainerDied","Data":"a6f0d37a791933965eb010150c0988434006bc639ee6ea88a596c3cd10bced1c"} Oct 31 21:34:00.172514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.172225 5188 scope.go:122] "RemoveContainer" containerID="a6f0d37a791933965eb010150c0988434006bc639ee6ea88a596c3cd10bced1c" Oct 31 21:34:00.173674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.173647 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:34:00.173784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.173724 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:34:00.173784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.173771 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:34:00.804178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.804142 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:34:00.805873 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.805834 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:34:00.826001 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:00.808025 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:34:01.234284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:01.233952 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" event={"ID":"96111e4f-31bc-4769-a00a-ddf36fa4e2af","Type":"ContainerStarted","Data":"58d84e002395f43f5559ec4c46453561b40d9f8f8d93d96221f10114daadcd05"} Oct 31 21:34:01.236204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:01.236142 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" event={"ID":"378f9a55-a031-406d-bffc-ccc689ac4f29","Type":"ContainerStarted","Data":"0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424"} Oct 31 21:34:01.859165 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:01.859125 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:34:01.860382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:01.860355 5188 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:34:01.860526 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:01.860415 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 21:34:02.041902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:02.041861 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz"] Oct 31 21:34:02.047639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:02.047603 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ea69b3c-h8fsz"] Oct 31 21:34:02.093266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:02.092846 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:34:02.093266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:02.092996 5188 patch_prober.go:29] interesting pod/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 21:34:02.093266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:02.093045 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 21:34:02.715345 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:34:02.708019 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:34:02.859053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:02.859001 5188 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:34:02.859242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:02.859083 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 21:34:03.095689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:03.094767 5188 patch_prober.go:29] interesting pod/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 21:34:03.095689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:03.094834 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 21:34:03.514348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:03.513536 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb874929-a459-40a7-99a4-31769e515dcd" path="/var/lib/kubelet/pods/cb874929-a459-40a7-99a4-31769e515dcd/volumes" Oct 31 21:34:03.699347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:03.698735 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:34:03.859578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:03.859488 5188 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:34:03.859578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:03.859548 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 21:34:04.087896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:04.087856 5188 patch_prober.go:29] interesting pod/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 21:34:04.088211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:04.088167 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 21:34:04.712473 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:04.712431 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:34:04.858593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:04.858560 5188 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:34:04.858811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:04.858633 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 21:34:05.092212 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.092106 5188 patch_prober.go:29] interesting pod/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 21:34:05.092858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.092823 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 21:34:05.318303 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.318255 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh"] Oct 31 21:34:05.336242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.336195 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.348010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.347917 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh"] Oct 31 21:34:05.351651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.351575 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:34:05.360811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.360777 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:34:05.422347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.421812 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:34:05.422347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.421911 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:34:05.461392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.461345 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f51fc3-z28vh\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.461603 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.461442 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f51fc3-z28vh\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.461753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.461729 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrw2\" (UniqueName: \"kubernetes.io/projected/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-kube-api-access-dlrw2\") pod \"maas-api-revoke-sub-56f51fc3-z28vh\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.563255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.562895 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrw2\" (UniqueName: \"kubernetes.io/projected/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-kube-api-access-dlrw2\") pod \"maas-api-revoke-sub-56f51fc3-z28vh\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.563255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.563081 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f51fc3-z28vh\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.563255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.563130 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f51fc3-z28vh\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.563785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.563766 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f51fc3-z28vh\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.566251 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.566229 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f51fc3-z28vh\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.609036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.607416 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrw2\" (UniqueName: \"kubernetes.io/projected/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-kube-api-access-dlrw2\") pod \"maas-api-revoke-sub-56f51fc3-z28vh\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.659810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.659751 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:05.859385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.858780 5188 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:34:05.859385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:05.858852 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 21:34:06.105349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:06.102771 5188 patch_prober.go:29] interesting pod/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 31 21:34:06.105349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:06.102839 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 31 21:34:06.143785 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:34:06.143734 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1fc46c_dbbc_43c6_9fc4_920caf037cf1.slice/crio-317c69a66ea6ea3a0c97041ae63dfb36564c597dbf08eff22e23882275b39a74 WatchSource:0}: Error finding container 317c69a66ea6ea3a0c97041ae63dfb36564c597dbf08eff22e23882275b39a74: Status 404 returned error can't find the container with id 317c69a66ea6ea3a0c97041ae63dfb36564c597dbf08eff22e23882275b39a74 Oct 31 21:34:06.149613 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:06.149586 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh"] Oct 31 21:34:06.345123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:06.345069 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" event={"ID":"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1","Type":"ContainerStarted","Data":"317c69a66ea6ea3a0c97041ae63dfb36564c597dbf08eff22e23882275b39a74"} Oct 31 21:34:06.864027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:06.863993 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:34:06.865194 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:06.865164 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:34:07.092698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:07.092620 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:34:07.094092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:07.094066 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:34:07.356706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:07.356606 5188 generic.go:356] "Generic (PLEG): container finished" podID="ed1fc46c-dbbc-43c6-9fc4-920caf037cf1" containerID="27f0d8c47bc8155c77d315c0059b93d9efeb9c5a3fe95888baf17fab73058407" exitCode=0 Oct 31 21:34:07.357171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:07.356696 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" event={"ID":"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1","Type":"ContainerDied","Data":"27f0d8c47bc8155c77d315c0059b93d9efeb9c5a3fe95888baf17fab73058407"} Oct 31 21:34:08.031710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.031674 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz"] Oct 31 21:34:08.042147 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.042104 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32af2159-ppsbz"] Oct 31 21:34:08.048293 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.048217 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt"] Oct 31 21:34:08.065197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.065158 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65eb6c27-xxwlt"] Oct 31 21:34:08.548757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.548728 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:08.596819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.596781 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-kube-api-access-dlrw2\" (UniqueName: \"kubernetes.io/projected/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-kube-api-access-dlrw2\") pod \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " Oct 31 21:34:08.597030 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.597001 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-xks-ca\") pod \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " Oct 31 21:34:08.597186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.597166 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-openshift-ca\") pod \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\" (UID: \"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1\") " Oct 31 21:34:08.598129 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.597679 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-openshift-ca" pod "ed1fc46c-dbbc-43c6-9fc4-920caf037cf1" (UID: "ed1fc46c-dbbc-43c6-9fc4-920caf037cf1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:34:08.598129 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.598084 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:34:08.600521 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.600491 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-kube-api-access-dlrw2" pod "ed1fc46c-dbbc-43c6-9fc4-920caf037cf1" (UID: "ed1fc46c-dbbc-43c6-9fc4-920caf037cf1"). InnerVolumeSpecName "kube-api-access-dlrw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:34:08.602173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.602107 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-xks-ca" pod "ed1fc46c-dbbc-43c6-9fc4-920caf037cf1" (UID: "ed1fc46c-dbbc-43c6-9fc4-920caf037cf1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:34:08.700338 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.700227 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:34:08.700338 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:08.700260 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrw2\" (UniqueName: \"kubernetes.io/projected/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1-kube-api-access-dlrw2\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:34:09.387212 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:09.387173 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" Oct 31 21:34:09.387212 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:09.387190 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh" event={"ID":"ed1fc46c-dbbc-43c6-9fc4-920caf037cf1","Type":"ContainerDied","Data":"317c69a66ea6ea3a0c97041ae63dfb36564c597dbf08eff22e23882275b39a74"} Oct 31 21:34:09.387538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:09.387287 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317c69a66ea6ea3a0c97041ae63dfb36564c597dbf08eff22e23882275b39a74" Oct 31 21:34:09.531357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:09.529643 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e00c871-5272-4032-90b1-00a6e69c9909" path="/var/lib/kubelet/pods/5e00c871-5272-4032-90b1-00a6e69c9909/volumes" Oct 31 21:34:09.531357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:09.530500 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b105f6f6-4640-4e02-8edb-45f9cd86f092" path="/var/lib/kubelet/pods/b105f6f6-4640-4e02-8edb-45f9cd86f092/volumes" Oct 31 21:34:12.034869 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:12.034830 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn"] Oct 31 21:34:12.044086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:12.044054 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca46659-sjtrn"] Oct 31 21:34:13.476593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:13.476554 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9e1334-4113-43bd-b348-5fd17bc24f6a" path="/var/lib/kubelet/pods/3f9e1334-4113-43bd-b348-5fd17bc24f6a/volumes" Oct 31 21:34:23.039371 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:23.039288 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl"] Oct 31 21:34:23.045736 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:23.045699 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f4ad3fa-qtddl"] Oct 31 21:34:23.484382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:23.480710 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7866c78e-aaae-4fdb-948e-9facecee1434" path="/var/lib/kubelet/pods/7866c78e-aaae-4fdb-948e-9facecee1434/volumes" Oct 31 21:34:24.042639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:24.041856 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb"] Oct 31 21:34:24.060251 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:24.060207 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf5804-ndcdb"] Oct 31 21:34:25.476821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:25.476785 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bde2fde-9142-40c6-bebc-22e38137ae60" path="/var/lib/kubelet/pods/8bde2fde-9142-40c6-bebc-22e38137ae60/volumes" Oct 31 21:34:26.923393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:26.923355 5188 generic.go:356] "Generic (PLEG): container finished" podID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerID="827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d" exitCode=0 Oct 31 21:34:26.923902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:26.923424 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" event={"ID":"ae718dc5-7881-4f3f-a98d-a517cefd0ab4","Type":"ContainerDied","Data":"827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d"} Oct 31 21:34:26.923976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:26.923950 5188 scope.go:122] "RemoveContainer" containerID="827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d" Oct 31 21:34:26.925650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:26.925618 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:34:26.925771 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:26.925703 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:34:26.925771 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:26.925743 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:34:27.935720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:27.935689 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" event={"ID":"ae718dc5-7881-4f3f-a98d-a517cefd0ab4","Type":"ContainerStarted","Data":"439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d"} Oct 31 21:34:28.678581 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:28.678537 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:34:28.696529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:28.696479 5188 patch_prober.go:29] interesting pod/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 21:34:28.696853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:28.696806 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 21:34:29.687352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:29.686463 5188 patch_prober.go:29] interesting pod/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 21:34:29.687352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:29.686551 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 21:34:30.678433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:30.678387 5188 patch_prober.go:29] interesting pod/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 21:34:30.678741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:30.678711 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 21:34:31.678819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:31.678775 5188 patch_prober.go:29] interesting pod/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 21:34:31.679519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:31.679482 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 21:34:32.648589 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:34:32.648547 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:34:32.691480 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:32.691445 5188 patch_prober.go:29] interesting pod/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 31 21:34:32.693934 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:32.693896 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 31 21:34:32.965479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:32.965375 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:34:33.684235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:33.684197 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:34:33.685845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:33.685814 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:34:37.029489 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.029440 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9"] Oct 31 21:34:37.043837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.043801 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ed1fc46c-dbbc-43c6-9fc4-920caf037cf1" containerName="revoke-keys" Oct 31 21:34:37.044095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.044076 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1fc46c-dbbc-43c6-9fc4-920caf037cf1" containerName="revoke-keys" Oct 31 21:34:37.044501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.044484 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1fc46c-dbbc-43c6-9fc4-920caf037cf1" containerName="revoke-keys" Oct 31 21:34:37.053649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.053614 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.060678 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.060645 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9"] Oct 31 21:34:37.087664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.082988 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:34:37.087664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.083175 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:34:37.087664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.083263 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:34:37.087664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.083350 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:34:37.145113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.145051 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305b981d-8395-442f-b810-72369cdd1807-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90db52-tczw9\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.145419 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.145402 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305b981d-8395-442f-b810-72369cdd1807-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90db52-tczw9\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.145576 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.145563 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fjbm\" (UniqueName: \"kubernetes.io/projected/305b981d-8395-442f-b810-72369cdd1807-kube-api-access-6fjbm\") pod \"maas-api-revoke-keys-e2e-api-a-90db52-tczw9\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.247150 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.247110 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305b981d-8395-442f-b810-72369cdd1807-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90db52-tczw9\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.247400 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.247222 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305b981d-8395-442f-b810-72369cdd1807-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90db52-tczw9\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.247400 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.247301 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fjbm\" (UniqueName: \"kubernetes.io/projected/305b981d-8395-442f-b810-72369cdd1807-kube-api-access-6fjbm\") pod \"maas-api-revoke-keys-e2e-api-a-90db52-tczw9\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.248093 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.248064 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305b981d-8395-442f-b810-72369cdd1807-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90db52-tczw9\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.249948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.249922 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305b981d-8395-442f-b810-72369cdd1807-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-90db52-tczw9\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.268686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.268653 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fjbm\" (UniqueName: \"kubernetes.io/projected/305b981d-8395-442f-b810-72369cdd1807-kube-api-access-6fjbm\") pod \"maas-api-revoke-keys-e2e-api-a-90db52-tczw9\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:37.409803 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:37.409768 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:38.266985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:38.266948 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9"] Oct 31 21:34:39.958094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:39.958047 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" event={"ID":"305b981d-8395-442f-b810-72369cdd1807","Type":"ContainerStarted","Data":"a94d93014c78cc4860f40f5eb5bff027150cb8197a720751358655a061e332ea"} Oct 31 21:34:41.060349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:41.059648 5188 generic.go:356] "Generic (PLEG): container finished" podID="305b981d-8395-442f-b810-72369cdd1807" containerID="0b935aa28384a22e385da0ffbc3526845d19574cc10268802ff7edb48d57a968" exitCode=0 Oct 31 21:34:41.060349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:41.060051 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" event={"ID":"305b981d-8395-442f-b810-72369cdd1807","Type":"ContainerDied","Data":"0b935aa28384a22e385da0ffbc3526845d19574cc10268802ff7edb48d57a968"} Oct 31 21:34:42.297304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.297270 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:42.426415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.426348 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305b981d-8395-442f-b810-72369cdd1807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305b981d-8395-442f-b810-72369cdd1807-maas-api-openshift-ca\") pod \"305b981d-8395-442f-b810-72369cdd1807\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " Oct 31 21:34:42.426415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.426417 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305b981d-8395-442f-b810-72369cdd1807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305b981d-8395-442f-b810-72369cdd1807-maas-api-xks-ca\") pod \"305b981d-8395-442f-b810-72369cdd1807\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " Oct 31 21:34:42.426708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.426644 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305b981d-8395-442f-b810-72369cdd1807-kube-api-access-6fjbm\" (UniqueName: \"kubernetes.io/projected/305b981d-8395-442f-b810-72369cdd1807-kube-api-access-6fjbm\") pod \"305b981d-8395-442f-b810-72369cdd1807\" (UID: \"305b981d-8395-442f-b810-72369cdd1807\") " Oct 31 21:34:42.426832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.426800 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305b981d-8395-442f-b810-72369cdd1807-maas-api-openshift-ca" pod "305b981d-8395-442f-b810-72369cdd1807" (UID: "305b981d-8395-442f-b810-72369cdd1807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:34:42.427192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.427175 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305b981d-8395-442f-b810-72369cdd1807-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:34:42.428607 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.428578 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305b981d-8395-442f-b810-72369cdd1807-maas-api-xks-ca" pod "305b981d-8395-442f-b810-72369cdd1807" (UID: "305b981d-8395-442f-b810-72369cdd1807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:34:42.428747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.428724 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305b981d-8395-442f-b810-72369cdd1807-kube-api-access-6fjbm" pod "305b981d-8395-442f-b810-72369cdd1807" (UID: "305b981d-8395-442f-b810-72369cdd1807"). InnerVolumeSpecName "kube-api-access-6fjbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:34:42.528682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.528632 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fjbm\" (UniqueName: \"kubernetes.io/projected/305b981d-8395-442f-b810-72369cdd1807-kube-api-access-6fjbm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:34:42.528682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:42.528670 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305b981d-8395-442f-b810-72369cdd1807-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:34:43.082046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.082011 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" Oct 31 21:34:43.082236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.082009 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9" event={"ID":"305b981d-8395-442f-b810-72369cdd1807","Type":"ContainerDied","Data":"a94d93014c78cc4860f40f5eb5bff027150cb8197a720751358655a061e332ea"} Oct 31 21:34:43.082236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.082164 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a94d93014c78cc4860f40f5eb5bff027150cb8197a720751358655a061e332ea" Oct 31 21:34:43.555544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.555491 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff"] Oct 31 21:34:43.556781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.556750 5188 cpu_manager.go:414] "containerMap: removing container" podUID="305b981d-8395-442f-b810-72369cdd1807" containerName="revoke-keys" Oct 31 21:34:43.557002 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.556980 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305b981d-8395-442f-b810-72369cdd1807" containerName="revoke-keys" Oct 31 21:34:43.557478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.557450 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="305b981d-8395-442f-b810-72369cdd1807" containerName="revoke-keys" Oct 31 21:34:43.562140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.562118 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.580421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.578416 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:34:43.580421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.579201 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:34:43.582725 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.582697 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff"] Oct 31 21:34:43.584094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.584050 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:34:43.584248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.584148 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:34:43.644588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.644557 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1420c1-fmjff\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.644771 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.644689 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1420c1-fmjff\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.644832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.644818 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6wmn\" (UniqueName: \"kubernetes.io/projected/0ca24a05-df1d-4c45-beda-9b8c9ae69779-kube-api-access-n6wmn\") pod \"maas-api-revoke-sub-1b1420c1-fmjff\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.746730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.746695 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1420c1-fmjff\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.746904 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.746890 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6wmn\" (UniqueName: \"kubernetes.io/projected/0ca24a05-df1d-4c45-beda-9b8c9ae69779-kube-api-access-n6wmn\") pod \"maas-api-revoke-sub-1b1420c1-fmjff\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.747008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.746991 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1420c1-fmjff\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.747827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.747802 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1420c1-fmjff\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.748950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.748925 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1420c1-fmjff\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.768413 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.768385 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6wmn\" (UniqueName: \"kubernetes.io/projected/0ca24a05-df1d-4c45-beda-9b8c9ae69779-kube-api-access-n6wmn\") pod \"maas-api-revoke-sub-1b1420c1-fmjff\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:43.885069 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:43.885029 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:44.199131 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:34:44.199084 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca24a05_df1d_4c45_beda_9b8c9ae69779.slice/crio-2bfd4d838018eb3afb3ec755fd52fefa1db9a47aa9be310e0c456326221ec3df WatchSource:0}: Error finding container 2bfd4d838018eb3afb3ec755fd52fefa1db9a47aa9be310e0c456326221ec3df: Status 404 returned error can't find the container with id 2bfd4d838018eb3afb3ec755fd52fefa1db9a47aa9be310e0c456326221ec3df Oct 31 21:34:44.199640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:44.199617 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff"] Oct 31 21:34:45.121207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:45.121158 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" event={"ID":"0ca24a05-df1d-4c45-beda-9b8c9ae69779","Type":"ContainerStarted","Data":"d50039481a0edc1bd80cd4c5ee5e4623819f6aef53257afa4b2adc3714a644a7"} Oct 31 21:34:45.121750 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:45.121286 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" event={"ID":"0ca24a05-df1d-4c45-beda-9b8c9ae69779","Type":"ContainerStarted","Data":"2bfd4d838018eb3afb3ec755fd52fefa1db9a47aa9be310e0c456326221ec3df"} Oct 31 21:34:45.168077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:45.166829 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" podStartSLOduration=2.166803371 podStartE2EDuration="2.166803371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:34:45.156334364 +0000 UTC m=+1131.849291750" watchObservedRunningTime="2026-10-31 21:34:45.166803371 +0000 UTC m=+1131.859760773" Oct 31 21:34:46.133236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:46.133196 5188 generic.go:356] "Generic (PLEG): container finished" podID="0ca24a05-df1d-4c45-beda-9b8c9ae69779" containerID="d50039481a0edc1bd80cd4c5ee5e4623819f6aef53257afa4b2adc3714a644a7" exitCode=0 Oct 31 21:34:46.133817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:46.133272 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" event={"ID":"0ca24a05-df1d-4c45-beda-9b8c9ae69779","Type":"ContainerDied","Data":"d50039481a0edc1bd80cd4c5ee5e4623819f6aef53257afa4b2adc3714a644a7"} Oct 31 21:34:47.339695 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.339662 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:47.408316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.408215 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-openshift-ca\") pod \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " Oct 31 21:34:47.408491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.408375 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca24a05-df1d-4c45-beda-9b8c9ae69779-kube-api-access-n6wmn\" (UniqueName: \"kubernetes.io/projected/0ca24a05-df1d-4c45-beda-9b8c9ae69779-kube-api-access-n6wmn\") pod \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " Oct 31 21:34:47.408491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.408463 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-xks-ca\") pod \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\" (UID: \"0ca24a05-df1d-4c45-beda-9b8c9ae69779\") " Oct 31 21:34:47.408758 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.408728 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-openshift-ca" pod "0ca24a05-df1d-4c45-beda-9b8c9ae69779" (UID: "0ca24a05-df1d-4c45-beda-9b8c9ae69779"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:34:47.409189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.409161 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:34:47.411532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.411482 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-xks-ca" pod "0ca24a05-df1d-4c45-beda-9b8c9ae69779" (UID: "0ca24a05-df1d-4c45-beda-9b8c9ae69779"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:34:47.411713 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.411680 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca24a05-df1d-4c45-beda-9b8c9ae69779-kube-api-access-n6wmn" pod "0ca24a05-df1d-4c45-beda-9b8c9ae69779" (UID: "0ca24a05-df1d-4c45-beda-9b8c9ae69779"). InnerVolumeSpecName "kube-api-access-n6wmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:34:47.510782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.510736 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6wmn\" (UniqueName: \"kubernetes.io/projected/0ca24a05-df1d-4c45-beda-9b8c9ae69779-kube-api-access-n6wmn\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:34:47.510782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:47.510782 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ca24a05-df1d-4c45-beda-9b8c9ae69779-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:34:48.155049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:48.155014 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" event={"ID":"0ca24a05-df1d-4c45-beda-9b8c9ae69779","Type":"ContainerDied","Data":"2bfd4d838018eb3afb3ec755fd52fefa1db9a47aa9be310e0c456326221ec3df"} Oct 31 21:34:48.155049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:48.155016 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff" Oct 31 21:34:48.155296 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:48.155067 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bfd4d838018eb3afb3ec755fd52fefa1db9a47aa9be310e0c456326221ec3df" Oct 31 21:34:54.427074 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:54.427042 5188 scope.go:122] "RemoveContainer" containerID="276d2094db1437e056d72a6cfe389d13b7cea159194c8f43c1b2e866a6776b46" Oct 31 21:34:54.439296 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:34:54.439270 5188 scope.go:122] "RemoveContainer" containerID="41466d4e6b35be3c810b024d6ccd62011b120271be4d5f55afa15e94711f6c69" Oct 31 21:35:02.541306 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:35:02.541261 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:35:06.361545 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.361502 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw"] Oct 31 21:35:06.362242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.362225 5188 cpu_manager.go:414] "containerMap: removing container" podUID="0ca24a05-df1d-4c45-beda-9b8c9ae69779" containerName="revoke-keys" Oct 31 21:35:06.362292 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.362244 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca24a05-df1d-4c45-beda-9b8c9ae69779" containerName="revoke-keys" Oct 31 21:35:06.362408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.362398 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca24a05-df1d-4c45-beda-9b8c9ae69779" containerName="revoke-keys" Oct 31 21:35:06.366852 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.366830 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.397721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.390622 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:35:06.397721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.391775 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:35:06.397721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.392594 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:35:06.397721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.392713 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:35:06.401570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.401536 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw"] Oct 31 21:35:06.496429 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.496379 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brk8c\" (UniqueName: \"kubernetes.io/projected/8b8be49f-38b4-4410-8786-bea8477b11ec-kube-api-access-brk8c\") pod \"maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.496625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.496504 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.496723 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.496701 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.598538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.598495 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.598862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.598840 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brk8c\" (UniqueName: \"kubernetes.io/projected/8b8be49f-38b4-4410-8786-bea8477b11ec-kube-api-access-brk8c\") pod \"maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.598980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.598899 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.599256 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.599234 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.601257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.601235 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.620071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.619993 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brk8c\" (UniqueName: \"kubernetes.io/projected/8b8be49f-38b4-4410-8786-bea8477b11ec-kube-api-access-brk8c\") pod \"maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:06.706952 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:06.706908 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:07.049038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:07.049005 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw"] Oct 31 21:35:07.354100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:07.354010 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" event={"ID":"8b8be49f-38b4-4410-8786-bea8477b11ec","Type":"ContainerStarted","Data":"8401e39000766511a30631a79858024b239c837fd8fe61d20934fe29589a4c39"} Oct 31 21:35:08.366128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:08.366092 5188 generic.go:356] "Generic (PLEG): container finished" podID="8b8be49f-38b4-4410-8786-bea8477b11ec" containerID="12558d84e09fd46c05ef80f5187a8c8472ee605145854a93cf7f2648ecc2d85d" exitCode=0 Oct 31 21:35:08.366652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:08.366166 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" event={"ID":"8b8be49f-38b4-4410-8786-bea8477b11ec","Type":"ContainerDied","Data":"12558d84e09fd46c05ef80f5187a8c8472ee605145854a93cf7f2648ecc2d85d"} Oct 31 21:35:09.577747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.577717 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:09.675684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.675577 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-xks-ca\") pod \"8b8be49f-38b4-4410-8786-bea8477b11ec\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " Oct 31 21:35:09.676016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.675989 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8be49f-38b4-4410-8786-bea8477b11ec-kube-api-access-brk8c\" (UniqueName: \"kubernetes.io/projected/8b8be49f-38b4-4410-8786-bea8477b11ec-kube-api-access-brk8c\") pod \"8b8be49f-38b4-4410-8786-bea8477b11ec\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " Oct 31 21:35:09.676267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.676251 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-openshift-ca\") pod \"8b8be49f-38b4-4410-8786-bea8477b11ec\" (UID: \"8b8be49f-38b4-4410-8786-bea8477b11ec\") " Oct 31 21:35:09.676739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.676708 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-openshift-ca" pod "8b8be49f-38b4-4410-8786-bea8477b11ec" (UID: "8b8be49f-38b4-4410-8786-bea8477b11ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:35:09.677287 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.677245 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:09.679272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.679239 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-xks-ca" pod "8b8be49f-38b4-4410-8786-bea8477b11ec" (UID: "8b8be49f-38b4-4410-8786-bea8477b11ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:35:09.679421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.679400 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8be49f-38b4-4410-8786-bea8477b11ec-kube-api-access-brk8c" pod "8b8be49f-38b4-4410-8786-bea8477b11ec" (UID: "8b8be49f-38b4-4410-8786-bea8477b11ec"). InnerVolumeSpecName "kube-api-access-brk8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:09.778956 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.778910 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8be49f-38b4-4410-8786-bea8477b11ec-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:09.778956 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:09.778947 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brk8c\" (UniqueName: \"kubernetes.io/projected/8b8be49f-38b4-4410-8786-bea8477b11ec-kube-api-access-brk8c\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:10.388293 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:10.388249 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" event={"ID":"8b8be49f-38b4-4410-8786-bea8477b11ec","Type":"ContainerDied","Data":"8401e39000766511a30631a79858024b239c837fd8fe61d20934fe29589a4c39"} Oct 31 21:35:10.388843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:10.388316 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8401e39000766511a30631a79858024b239c837fd8fe61d20934fe29589a4c39" Oct 31 21:35:10.388843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:10.388376 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw" Oct 31 21:35:17.504505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:17.504463 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-294zx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:35:17.505175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:17.505119 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:35:18.731628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:18.731581 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8"] Oct 31 21:35:19.605340 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.605288 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n"] Oct 31 21:35:19.606766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.606721 5188 cpu_manager.go:414] "containerMap: removing container" podUID="8b8be49f-38b4-4410-8786-bea8477b11ec" containerName="revoke-keys" Oct 31 21:35:19.606960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.606945 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8be49f-38b4-4410-8786-bea8477b11ec" containerName="revoke-keys" Oct 31 21:35:19.607269 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.607255 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8be49f-38b4-4410-8786-bea8477b11ec" containerName="revoke-keys" Oct 31 21:35:19.612810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.612770 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.626239 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.626204 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:35:19.628432 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.628366 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n"] Oct 31 21:35:19.635663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.635637 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:35:19.635949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.635935 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:35:19.636097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.636085 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:35:19.807702 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.807659 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-59z7n\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.808097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.807811 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-59z7n\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.808097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.807890 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztrp\" (UniqueName: \"kubernetes.io/projected/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-kube-api-access-2ztrp\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-59z7n\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.909717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.909670 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-59z7n\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.909948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.909800 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-59z7n\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.909948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.909864 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztrp\" (UniqueName: \"kubernetes.io/projected/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-kube-api-access-2ztrp\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-59z7n\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.910599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.910572 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-59z7n\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.912481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.912453 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-59z7n\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.931664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.931628 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztrp\" (UniqueName: \"kubernetes.io/projected/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-kube-api-access-2ztrp\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-59z7n\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:19.938219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:19.938193 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:20.257710 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:35:20.257672 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69ca93b_9bf1_441f_9df0_2e03823fdb2a.slice/crio-c94c16263d71369a8486909bc46a122066382346e10392cfc5bd8c07b077afb5 WatchSource:0}: Error finding container c94c16263d71369a8486909bc46a122066382346e10392cfc5bd8c07b077afb5: Status 404 returned error can't find the container with id c94c16263d71369a8486909bc46a122066382346e10392cfc5bd8c07b077afb5 Oct 31 21:35:20.258231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:20.258202 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n"] Oct 31 21:35:20.630316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:20.630280 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" event={"ID":"c69ca93b-9bf1-441f-9df0-2e03823fdb2a","Type":"ContainerStarted","Data":"c94c16263d71369a8486909bc46a122066382346e10392cfc5bd8c07b077afb5"} Oct 31 21:35:21.642598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:21.642558 5188 generic.go:356] "Generic (PLEG): container finished" podID="c69ca93b-9bf1-441f-9df0-2e03823fdb2a" containerID="dc2f92a0142bfbfeb15861cdbacacebebe31ec5db5974716893109a961d03454" exitCode=0 Oct 31 21:35:21.643042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:21.642648 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" event={"ID":"c69ca93b-9bf1-441f-9df0-2e03823fdb2a","Type":"ContainerDied","Data":"dc2f92a0142bfbfeb15861cdbacacebebe31ec5db5974716893109a961d03454"} Oct 31 21:35:22.820735 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.820700 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:22.857307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.857133 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-openshift-ca\") pod \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " Oct 31 21:35:22.857307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.857188 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-kube-api-access-2ztrp\" (UniqueName: \"kubernetes.io/projected/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-kube-api-access-2ztrp\") pod \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " Oct 31 21:35:22.857307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.857225 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-xks-ca\") pod \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\" (UID: \"c69ca93b-9bf1-441f-9df0-2e03823fdb2a\") " Oct 31 21:35:22.857710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.857662 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-openshift-ca" pod "c69ca93b-9bf1-441f-9df0-2e03823fdb2a" (UID: "c69ca93b-9bf1-441f-9df0-2e03823fdb2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:35:22.858798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.858105 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:22.860551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.860520 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-xks-ca" pod "c69ca93b-9bf1-441f-9df0-2e03823fdb2a" (UID: "c69ca93b-9bf1-441f-9df0-2e03823fdb2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:35:22.861203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.861172 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-kube-api-access-2ztrp" pod "c69ca93b-9bf1-441f-9df0-2e03823fdb2a" (UID: "c69ca93b-9bf1-441f-9df0-2e03823fdb2a"). InnerVolumeSpecName "kube-api-access-2ztrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:22.960171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.960067 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztrp\" (UniqueName: \"kubernetes.io/projected/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-kube-api-access-2ztrp\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:22.960171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:22.960109 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c69ca93b-9bf1-441f-9df0-2e03823fdb2a-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:23.664132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:23.664099 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" Oct 31 21:35:23.664132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:23.664123 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n" event={"ID":"c69ca93b-9bf1-441f-9df0-2e03823fdb2a","Type":"ContainerDied","Data":"c94c16263d71369a8486909bc46a122066382346e10392cfc5bd8c07b077afb5"} Oct 31 21:35:23.664426 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:23.664181 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c94c16263d71369a8486909bc46a122066382346e10392cfc5bd8c07b077afb5" Oct 31 21:35:23.733760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:23.733699 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" podUID="8b0f6229-c18a-46ee-b93b-5475367bee46" containerName="payload-pre-processing" containerID="cri-o://84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b" gracePeriod=25 Oct 31 21:35:24.094228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.093679 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:35:24.188150 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.188102 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0f6229-c18a-46ee-b93b-5475367bee46-kube-api-access-mnnlr\" (UniqueName: \"kubernetes.io/projected/8b0f6229-c18a-46ee-b93b-5475367bee46-kube-api-access-mnnlr\") pod \"8b0f6229-c18a-46ee-b93b-5475367bee46\" (UID: \"8b0f6229-c18a-46ee-b93b-5475367bee46\") " Oct 31 21:35:24.188378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.188223 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0f6229-c18a-46ee-b93b-5475367bee46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b0f6229-c18a-46ee-b93b-5475367bee46-plugins-config-volume\") pod \"8b0f6229-c18a-46ee-b93b-5475367bee46\" (UID: \"8b0f6229-c18a-46ee-b93b-5475367bee46\") " Oct 31 21:35:24.188766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.188729 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0f6229-c18a-46ee-b93b-5475367bee46-plugins-config-volume" pod "8b0f6229-c18a-46ee-b93b-5475367bee46" (UID: "8b0f6229-c18a-46ee-b93b-5475367bee46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:35:24.189206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.189178 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b0f6229-c18a-46ee-b93b-5475367bee46-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:24.190521 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.190493 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0f6229-c18a-46ee-b93b-5475367bee46-kube-api-access-mnnlr" pod "8b0f6229-c18a-46ee-b93b-5475367bee46" (UID: "8b0f6229-c18a-46ee-b93b-5475367bee46"). InnerVolumeSpecName "kube-api-access-mnnlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:24.290902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.290806 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnnlr\" (UniqueName: \"kubernetes.io/projected/8b0f6229-c18a-46ee-b93b-5475367bee46-kube-api-access-mnnlr\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:24.695172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.695137 5188 generic.go:356] "Generic (PLEG): container finished" podID="8b0f6229-c18a-46ee-b93b-5475367bee46" containerID="84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b" exitCode=0 Oct 31 21:35:24.695414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.695189 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" Oct 31 21:35:24.695414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.695217 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" event={"ID":"8b0f6229-c18a-46ee-b93b-5475367bee46","Type":"ContainerDied","Data":"84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b"} Oct 31 21:35:24.695414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.695284 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8" event={"ID":"8b0f6229-c18a-46ee-b93b-5475367bee46","Type":"ContainerDied","Data":"243dd5aa93f01e9a5738d30acaba50249cd4ea33f94e5f3850bf1c50c0247f7c"} Oct 31 21:35:24.695414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.695313 5188 scope.go:122] "RemoveContainer" containerID="84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b" Oct 31 21:35:24.712357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.712335 5188 scope.go:122] "RemoveContainer" containerID="84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b" Oct 31 21:35:24.712737 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:35:24.712716 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b\": container with ID starting with 84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b not found: ID does not exist" containerID="84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b" Oct 31 21:35:24.712803 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.712747 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b"} err="failed to get container status \"84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b\": rpc error: code = NotFound desc = could not find container \"84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b\": container with ID starting with 84e757aac0592d19dd198821dd82ec1a2647f844a5204c42ffb14727c95a8c2b not found: ID does not exist" Oct 31 21:35:24.746550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.746512 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8"] Oct 31 21:35:24.755676 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:24.755638 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7ac3c8-674d8f78bc-tdlt8"] Oct 31 21:35:25.476761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:25.476723 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0f6229-c18a-46ee-b93b-5475367bee46" path="/var/lib/kubelet/pods/8b0f6229-c18a-46ee-b93b-5475367bee46/volumes" Oct 31 21:35:26.198400 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:26.198353 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9"] Oct 31 21:35:26.207342 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:26.207295 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-90db52-tczw9"] Oct 31 21:35:27.474532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:27.474499 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305b981d-8395-442f-b810-72369cdd1807" path="/var/lib/kubelet/pods/305b981d-8395-442f-b810-72369cdd1807/volumes" Oct 31 21:35:32.539184 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:35:32.539133 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:35:33.968746 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:33.968712 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz"] Oct 31 21:35:33.969607 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:33.969570 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" containerID="cri-o://0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424" gracePeriod=30 Oct 31 21:35:34.547376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.547337 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924"] Oct 31 21:35:34.548187 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.548167 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c69ca93b-9bf1-441f-9df0-2e03823fdb2a" containerName="revoke-keys" Oct 31 21:35:34.548264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.548189 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69ca93b-9bf1-441f-9df0-2e03823fdb2a" containerName="revoke-keys" Oct 31 21:35:34.548264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.548222 5188 cpu_manager.go:414] "containerMap: removing container" podUID="8b0f6229-c18a-46ee-b93b-5475367bee46" containerName="payload-pre-processing" Oct 31 21:35:34.548264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.548232 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0f6229-c18a-46ee-b93b-5475367bee46" containerName="payload-pre-processing" Oct 31 21:35:34.548508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.548363 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69ca93b-9bf1-441f-9df0-2e03823fdb2a" containerName="revoke-keys" Oct 31 21:35:34.548508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.548378 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0f6229-c18a-46ee-b93b-5475367bee46" containerName="payload-pre-processing" Oct 31 21:35:34.553999 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.553973 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.571938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.571896 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924"] Oct 31 21:35:34.576611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.576576 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:35:34.576845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.576629 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:35:34.600883 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.600841 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:35:34.600883 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.600863 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:35:34.712619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.712572 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa31abc-9895-4597-9603-90863762aae0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-371531-g5924\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.712869 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.712824 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa31abc-9895-4597-9603-90863762aae0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-371531-g5924\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.712938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.712890 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzjx\" (UniqueName: \"kubernetes.io/projected/daa31abc-9895-4597-9603-90863762aae0-kube-api-access-jjzjx\") pod \"maas-api-revoke-keys-e2e-api-b-371531-g5924\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.814901 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.814806 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzjx\" (UniqueName: \"kubernetes.io/projected/daa31abc-9895-4597-9603-90863762aae0-kube-api-access-jjzjx\") pod \"maas-api-revoke-keys-e2e-api-b-371531-g5924\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.815084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.814945 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa31abc-9895-4597-9603-90863762aae0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-371531-g5924\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.815160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.815093 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa31abc-9895-4597-9603-90863762aae0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-371531-g5924\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.815678 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.815654 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa31abc-9895-4597-9603-90863762aae0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-371531-g5924\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.817696 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.817668 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa31abc-9895-4597-9603-90863762aae0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-371531-g5924\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.836370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.836316 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzjx\" (UniqueName: \"kubernetes.io/projected/daa31abc-9895-4597-9603-90863762aae0-kube-api-access-jjzjx\") pod \"maas-api-revoke-keys-e2e-api-b-371531-g5924\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:34.877025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:34.876981 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:35.221176 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:35.221095 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924"] Oct 31 21:35:35.954221 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:35.954119 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" event={"ID":"daa31abc-9895-4597-9603-90863762aae0","Type":"ContainerStarted","Data":"09ff79b82f8af61395715aa7c7951f17e5eb98b864730cf38e1942a5d2c68a88"} Oct 31 21:35:35.954221 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:35.954190 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" event={"ID":"daa31abc-9895-4597-9603-90863762aae0","Type":"ContainerStarted","Data":"6ed97b3e091c290fcfbd68e79ff05b7727e75a9b7b89753cdbfe26fca6bcba7e"} Oct 31 21:35:35.992112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:35.992045 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" podStartSLOduration=1.9920305539999998 podStartE2EDuration="1.992030554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:35:35.989778081 +0000 UTC m=+1182.682735473" watchObservedRunningTime="2026-10-31 21:35:35.992030554 +0000 UTC m=+1182.684987948" Oct 31 21:35:37.034812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:37.034763 5188 generic.go:356] "Generic (PLEG): container finished" podID="daa31abc-9895-4597-9603-90863762aae0" containerID="09ff79b82f8af61395715aa7c7951f17e5eb98b864730cf38e1942a5d2c68a88" exitCode=0 Oct 31 21:35:37.035406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:37.034918 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" event={"ID":"daa31abc-9895-4597-9603-90863762aae0","Type":"ContainerDied","Data":"09ff79b82f8af61395715aa7c7951f17e5eb98b864730cf38e1942a5d2c68a88"} Oct 31 21:35:37.130468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:37.130180 5188 patch_prober.go:29] interesting pod/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:35:37.130468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:37.130249 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:35:38.225431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.225401 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:38.285485 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.285436 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa31abc-9895-4597-9603-90863762aae0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa31abc-9895-4597-9603-90863762aae0-maas-api-xks-ca\") pod \"daa31abc-9895-4597-9603-90863762aae0\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " Oct 31 21:35:38.285691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.285572 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa31abc-9895-4597-9603-90863762aae0-kube-api-access-jjzjx\" (UniqueName: \"kubernetes.io/projected/daa31abc-9895-4597-9603-90863762aae0-kube-api-access-jjzjx\") pod \"daa31abc-9895-4597-9603-90863762aae0\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " Oct 31 21:35:38.285916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.285892 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa31abc-9895-4597-9603-90863762aae0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa31abc-9895-4597-9603-90863762aae0-maas-api-openshift-ca\") pod \"daa31abc-9895-4597-9603-90863762aae0\" (UID: \"daa31abc-9895-4597-9603-90863762aae0\") " Oct 31 21:35:38.291361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.287813 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa31abc-9895-4597-9603-90863762aae0-maas-api-openshift-ca" pod "daa31abc-9895-4597-9603-90863762aae0" (UID: "daa31abc-9895-4597-9603-90863762aae0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:35:38.296550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.296453 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa31abc-9895-4597-9603-90863762aae0-maas-api-xks-ca" pod "daa31abc-9895-4597-9603-90863762aae0" (UID: "daa31abc-9895-4597-9603-90863762aae0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:35:38.305163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.305093 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa31abc-9895-4597-9603-90863762aae0-kube-api-access-jjzjx" pod "daa31abc-9895-4597-9603-90863762aae0" (UID: "daa31abc-9895-4597-9603-90863762aae0"). InnerVolumeSpecName "kube-api-access-jjzjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:38.388888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.388845 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa31abc-9895-4597-9603-90863762aae0-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:38.388888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.388887 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa31abc-9895-4597-9603-90863762aae0-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:38.388888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:38.388898 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjzjx\" (UniqueName: \"kubernetes.io/projected/daa31abc-9895-4597-9603-90863762aae0-kube-api-access-jjzjx\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:39.057974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.057942 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" Oct 31 21:35:39.057974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.057965 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924" event={"ID":"daa31abc-9895-4597-9603-90863762aae0","Type":"ContainerDied","Data":"6ed97b3e091c290fcfbd68e79ff05b7727e75a9b7b89753cdbfe26fca6bcba7e"} Oct 31 21:35:39.058209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.058017 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed97b3e091c290fcfbd68e79ff05b7727e75a9b7b89753cdbfe26fca6bcba7e" Oct 31 21:35:39.310714 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.310684 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:35:39.415398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.415363 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-credential-socket\") pod \"378f9a55-a031-406d-bffc-ccc689ac4f29\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " Oct 31 21:35:39.415623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.415435 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-certs\") pod \"378f9a55-a031-406d-bffc-ccc689ac4f29\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " Oct 31 21:35:39.415623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.415497 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-kube-api-access-7ww2s\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-kube-api-access-7ww2s\") pod \"378f9a55-a031-406d-bffc-ccc689ac4f29\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " Oct 31 21:35:39.415623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.415535 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-envoy\") pod \"378f9a55-a031-406d-bffc-ccc689ac4f29\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " Oct 31 21:35:39.415781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.415761 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378f9a55-a031-406d-bffc-ccc689ac4f29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378f9a55-a031-406d-bffc-ccc689ac4f29-istiod-ca-cert\") pod \"378f9a55-a031-406d-bffc-ccc689ac4f29\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " Oct 31 21:35:39.415841 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.415803 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-token\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-token\") pod \"378f9a55-a031-406d-bffc-ccc689ac4f29\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " Oct 31 21:35:39.415896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.415854 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-socket\") pod \"378f9a55-a031-406d-bffc-ccc689ac4f29\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " Oct 31 21:35:39.415951 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.415904 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-data\") pod \"378f9a55-a031-406d-bffc-ccc689ac4f29\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " Oct 31 21:35:39.415951 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.415942 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-podinfo\") pod \"378f9a55-a031-406d-bffc-ccc689ac4f29\" (UID: \"378f9a55-a031-406d-bffc-ccc689ac4f29\") " Oct 31 21:35:39.417629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.417355 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-credential-socket" pod "378f9a55-a031-406d-bffc-ccc689ac4f29" (UID: "378f9a55-a031-406d-bffc-ccc689ac4f29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:35:39.417629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.417583 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-data" pod "378f9a55-a031-406d-bffc-ccc689ac4f29" (UID: "378f9a55-a031-406d-bffc-ccc689ac4f29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:35:39.418113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.418064 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-socket" pod "378f9a55-a031-406d-bffc-ccc689ac4f29" (UID: "378f9a55-a031-406d-bffc-ccc689ac4f29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:35:39.418217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.418186 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378f9a55-a031-406d-bffc-ccc689ac4f29-istiod-ca-cert" pod "378f9a55-a031-406d-bffc-ccc689ac4f29" (UID: "378f9a55-a031-406d-bffc-ccc689ac4f29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:35:39.418644 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.418624 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-certs" pod "378f9a55-a031-406d-bffc-ccc689ac4f29" (UID: "378f9a55-a031-406d-bffc-ccc689ac4f29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:35:39.423095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.423067 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-kube-api-access-7ww2s" pod "378f9a55-a031-406d-bffc-ccc689ac4f29" (UID: "378f9a55-a031-406d-bffc-ccc689ac4f29"). InnerVolumeSpecName "kube-api-access-7ww2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:39.423506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.423473 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-envoy" pod "378f9a55-a031-406d-bffc-ccc689ac4f29" (UID: "378f9a55-a031-406d-bffc-ccc689ac4f29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:35:39.423506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.423485 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-token" pod "378f9a55-a031-406d-bffc-ccc689ac4f29" (UID: "378f9a55-a031-406d-bffc-ccc689ac4f29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:39.423506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.423476 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-podinfo" pod "378f9a55-a031-406d-bffc-ccc689ac4f29" (UID: "378f9a55-a031-406d-bffc-ccc689ac4f29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:35:39.518392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.518296 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/378f9a55-a031-406d-bffc-ccc689ac4f29-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:39.518392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.518344 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:39.518392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.518354 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:39.518392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.518362 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:39.518392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.518372 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:39.518392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.518380 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:39.518392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.518389 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:39.518392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.518398 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ww2s\" (UniqueName: \"kubernetes.io/projected/378f9a55-a031-406d-bffc-ccc689ac4f29-kube-api-access-7ww2s\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:39.518777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:39.518406 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/378f9a55-a031-406d-bffc-ccc689ac4f29-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:40.174386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.174309 5188 generic.go:356] "Generic (PLEG): container finished" podID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerID="0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424" exitCode=0 Oct 31 21:35:40.174861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.174831 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" event={"ID":"378f9a55-a031-406d-bffc-ccc689ac4f29","Type":"ContainerDied","Data":"0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424"} Oct 31 21:35:40.175058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.175038 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" event={"ID":"378f9a55-a031-406d-bffc-ccc689ac4f29","Type":"ContainerDied","Data":"9ad8f81359742fa8a3db74379097ac3023921cbb710401545731baa2e47c47ae"} Oct 31 21:35:40.175192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.175178 5188 scope.go:122] "RemoveContainer" containerID="0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424" Oct 31 21:35:40.175505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.175482 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz" Oct 31 21:35:40.199748 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.199716 5188 scope.go:122] "RemoveContainer" containerID="1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738" Oct 31 21:35:40.220135 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.220112 5188 scope.go:122] "RemoveContainer" containerID="0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424" Oct 31 21:35:40.220560 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:35:40.220533 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424\": container with ID starting with 0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424 not found: ID does not exist" containerID="0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424" Oct 31 21:35:40.220615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.220576 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424"} err="failed to get container status \"0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424\": rpc error: code = NotFound desc = could not find container \"0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424\": container with ID starting with 0f198ef52be8623581a62046c7274ee7823e0dbee60dc5820129ce94ed393424 not found: ID does not exist" Oct 31 21:35:40.220615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.220600 5188 scope.go:122] "RemoveContainer" containerID="1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738" Oct 31 21:35:40.220928 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:35:40.220894 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738\": container with ID starting with 1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738 not found: ID does not exist" containerID="1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738" Oct 31 21:35:40.221036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.220926 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738"} err="failed to get container status \"1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738\": rpc error: code = NotFound desc = could not find container \"1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738\": container with ID starting with 1bd01b410afafc0b0d5160edcb818b2b5c348c5eedcaf617ffbb6448e338e738 not found: ID does not exist" Oct 31 21:35:40.243979 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.243913 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz"] Oct 31 21:35:40.255598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:40.255557 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-90db52-openshift-default-f788579b9-xp2jz"] Oct 31 21:35:41.487090 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:41.486555 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" path="/var/lib/kubelet/pods/378f9a55-a031-406d-bffc-ccc689ac4f29/volumes" Oct 31 21:35:46.418959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.418096 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd"] Oct 31 21:35:46.419617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.419185 5188 cpu_manager.go:414] "containerMap: removing container" podUID="daa31abc-9895-4597-9603-90863762aae0" containerName="revoke-keys" Oct 31 21:35:46.419617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.419207 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa31abc-9895-4597-9603-90863762aae0" containerName="revoke-keys" Oct 31 21:35:46.419617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.419253 5188 cpu_manager.go:414] "containerMap: removing container" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" Oct 31 21:35:46.419617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.419262 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" Oct 31 21:35:46.419617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.419483 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa31abc-9895-4597-9603-90863762aae0" containerName="revoke-keys" Oct 31 21:35:46.419617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.419507 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" Oct 31 21:35:46.446234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.438792 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd"] Oct 31 21:35:46.446234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.438926 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.472683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.467024 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:35:46.473054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.473025 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:35:46.474783 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.473385 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:35:46.526380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.523086 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr9fp\" (UniqueName: \"kubernetes.io/projected/074515e8-bd43-4184-a4a1-ad009a26e61d-kube-api-access-rr9fp\") pod \"maas-api-revoke-sub-607c7327-fgttd\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.526380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.523234 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-607c7327-fgttd\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.526380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.523344 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-607c7327-fgttd\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.624773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.624734 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:35:46.626634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.626598 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr9fp\" (UniqueName: \"kubernetes.io/projected/074515e8-bd43-4184-a4a1-ad009a26e61d-kube-api-access-rr9fp\") pod \"maas-api-revoke-sub-607c7327-fgttd\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.627022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.626912 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-607c7327-fgttd\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.627285 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.627232 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-607c7327-fgttd\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.629194 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.628967 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-607c7327-fgttd\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.631385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.630888 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-607c7327-fgttd\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.649698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.649662 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr9fp\" (UniqueName: \"kubernetes.io/projected/074515e8-bd43-4184-a4a1-ad009a26e61d-kube-api-access-rr9fp\") pod \"maas-api-revoke-sub-607c7327-fgttd\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:46.886574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.886538 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw"] Oct 31 21:35:46.896389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.896282 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ac3c8-dwgnw"] Oct 31 21:35:46.931862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:46.931826 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:47.494813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:47.494783 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8be49f-38b4-4410-8786-bea8477b11ec" path="/var/lib/kubelet/pods/8b8be49f-38b4-4410-8786-bea8477b11ec/volumes" Oct 31 21:35:47.495611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:47.495590 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd"] Oct 31 21:35:48.031995 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:48.031903 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv"] Oct 31 21:35:48.041675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:48.041635 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f0f65e0-x7jxv"] Oct 31 21:35:48.506972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:48.506934 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" event={"ID":"074515e8-bd43-4184-a4a1-ad009a26e61d","Type":"ContainerStarted","Data":"a71e48cfb0fea0588fbc4e3ea6815e1d70aa01ead7b915dc2bbef179ae5ce263"} Oct 31 21:35:48.507545 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:48.507002 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" event={"ID":"074515e8-bd43-4184-a4a1-ad009a26e61d","Type":"ContainerStarted","Data":"2180d5b194e6838b5f1bb25d825d5c8e883e955be482adbaa57dd244c2eebffa"} Oct 31 21:35:48.542795 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:48.542724 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" podStartSLOduration=2.542709146 podStartE2EDuration="2.542709146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:35:48.542534288 +0000 UTC m=+1195.235491681" watchObservedRunningTime="2026-10-31 21:35:48.542709146 +0000 UTC m=+1195.235666538" Oct 31 21:35:49.476911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:49.476861 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa255a70-9477-4694-9f01-df562b2dbead" path="/var/lib/kubelet/pods/fa255a70-9477-4694-9f01-df562b2dbead/volumes" Oct 31 21:35:49.517878 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:49.517839 5188 generic.go:356] "Generic (PLEG): container finished" podID="074515e8-bd43-4184-a4a1-ad009a26e61d" containerID="a71e48cfb0fea0588fbc4e3ea6815e1d70aa01ead7b915dc2bbef179ae5ce263" exitCode=0 Oct 31 21:35:49.518371 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:49.517917 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" event={"ID":"074515e8-bd43-4184-a4a1-ad009a26e61d","Type":"ContainerDied","Data":"a71e48cfb0fea0588fbc4e3ea6815e1d70aa01ead7b915dc2bbef179ae5ce263"} Oct 31 21:35:50.711786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.711755 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:50.740771 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.740737 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074515e8-bd43-4184-a4a1-ad009a26e61d-kube-api-access-rr9fp\" (UniqueName: \"kubernetes.io/projected/074515e8-bd43-4184-a4a1-ad009a26e61d-kube-api-access-rr9fp\") pod \"074515e8-bd43-4184-a4a1-ad009a26e61d\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " Oct 31 21:35:50.740975 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.740960 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-xks-ca\") pod \"074515e8-bd43-4184-a4a1-ad009a26e61d\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " Oct 31 21:35:50.741022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.741012 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-openshift-ca\") pod \"074515e8-bd43-4184-a4a1-ad009a26e61d\" (UID: \"074515e8-bd43-4184-a4a1-ad009a26e61d\") " Oct 31 21:35:50.741442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.741409 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-openshift-ca" pod "074515e8-bd43-4184-a4a1-ad009a26e61d" (UID: "074515e8-bd43-4184-a4a1-ad009a26e61d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:35:50.741604 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.741584 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:50.743043 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.743020 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-xks-ca" pod "074515e8-bd43-4184-a4a1-ad009a26e61d" (UID: "074515e8-bd43-4184-a4a1-ad009a26e61d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:35:50.743125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.743068 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074515e8-bd43-4184-a4a1-ad009a26e61d-kube-api-access-rr9fp" pod "074515e8-bd43-4184-a4a1-ad009a26e61d" (UID: "074515e8-bd43-4184-a4a1-ad009a26e61d"). InnerVolumeSpecName "kube-api-access-rr9fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:50.843964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.843805 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr9fp\" (UniqueName: \"kubernetes.io/projected/074515e8-bd43-4184-a4a1-ad009a26e61d-kube-api-access-rr9fp\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:50.843964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:50.843850 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/074515e8-bd43-4184-a4a1-ad009a26e61d-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:51.538283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:51.538251 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" Oct 31 21:35:51.538492 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:51.538281 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd" event={"ID":"074515e8-bd43-4184-a4a1-ad009a26e61d","Type":"ContainerDied","Data":"2180d5b194e6838b5f1bb25d825d5c8e883e955be482adbaa57dd244c2eebffa"} Oct 31 21:35:51.538492 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:51.538364 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2180d5b194e6838b5f1bb25d825d5c8e883e955be482adbaa57dd244c2eebffa" Oct 31 21:35:53.729355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:53.729310 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:35:53.735705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:53.735673 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:35:54.499563 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:54.499517 5188 scope.go:122] "RemoveContainer" containerID="308e22d57f4a1c5a6c904614ad3eaddb0d6d3c58307c9e45519255759d554c59" Oct 31 21:35:54.518618 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:54.518542 5188 scope.go:122] "RemoveContainer" containerID="d1b69b1cdda1766bc7b7adbf07f7754b360855d6b5cd340f2f7da042834bed3f" Oct 31 21:35:54.531101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:54.531074 5188 scope.go:122] "RemoveContainer" containerID="f16436add7df7959112728734f2866f2de98919723f12e1136115f7baa2e8fd7" Oct 31 21:35:54.543038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:54.543010 5188 scope.go:122] "RemoveContainer" containerID="3d03f589aca066d0055f0a1b59cb980bc9f6ec7dc038480ab83469ca917eeda6" Oct 31 21:35:54.554743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:54.554715 5188 scope.go:122] "RemoveContainer" containerID="55238865cca5a67f8a82b6fc028b58a48dce586d6f56e4ab6dea73c0bae12518" Oct 31 21:35:54.567068 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:54.567037 5188 scope.go:122] "RemoveContainer" containerID="599a5fe0a34f9a95e678e0b8a1a4c2bf72dba961a25c3f7bd4d7626623ca8afd" Oct 31 21:35:54.920638 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:54.920600 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf"] Oct 31 21:35:54.921500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:54.921464 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" containerID="cri-o://439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d" gracePeriod=30 Oct 31 21:35:55.504166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.504126 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz"] Oct 31 21:35:55.505466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.505255 5188 cpu_manager.go:414] "containerMap: removing container" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" Oct 31 21:35:55.505466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.505283 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" Oct 31 21:35:55.505466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.505305 5188 cpu_manager.go:414] "containerMap: removing container" podUID="074515e8-bd43-4184-a4a1-ad009a26e61d" containerName="revoke-keys" Oct 31 21:35:55.505466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.505314 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074515e8-bd43-4184-a4a1-ad009a26e61d" containerName="revoke-keys" Oct 31 21:35:55.505784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.505554 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="378f9a55-a031-406d-bffc-ccc689ac4f29" containerName="istio-proxy" Oct 31 21:35:55.505784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.505574 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="074515e8-bd43-4184-a4a1-ad009a26e61d" containerName="revoke-keys" Oct 31 21:35:55.512085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.512056 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.530150 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.530113 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz"] Oct 31 21:35:55.531886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.531835 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:35:55.532062 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.531944 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:35:55.562345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.556028 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:35:55.562345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.556179 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:35:55.635749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.635708 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-248b29-stxvz\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.635972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.635911 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqpw5\" (UniqueName: \"kubernetes.io/projected/3c6373a6-6f55-4004-a7b1-66ada946c9be-kube-api-access-lqpw5\") pod \"maas-api-revoke-keys-e2e-api-b-248b29-stxvz\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.636040 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.635976 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-248b29-stxvz\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.737998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.737959 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-248b29-stxvz\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.738260 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.738064 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqpw5\" (UniqueName: \"kubernetes.io/projected/3c6373a6-6f55-4004-a7b1-66ada946c9be-kube-api-access-lqpw5\") pod \"maas-api-revoke-keys-e2e-api-b-248b29-stxvz\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.738260 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.738104 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-248b29-stxvz\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.738924 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.738889 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-248b29-stxvz\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.740694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.740667 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-248b29-stxvz\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.765781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.765699 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqpw5\" (UniqueName: \"kubernetes.io/projected/3c6373a6-6f55-4004-a7b1-66ada946c9be-kube-api-access-lqpw5\") pod \"maas-api-revoke-keys-e2e-api-b-248b29-stxvz\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:55.862400 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:55.862305 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:56.191773 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:35:56.191485 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6373a6_6f55_4004_a7b1_66ada946c9be.slice/crio-e99617ee2acfb31b9bb76392e4f7a01e30887a50e6dd2fc69bd57116d7337397 WatchSource:0}: Error finding container e99617ee2acfb31b9bb76392e4f7a01e30887a50e6dd2fc69bd57116d7337397: Status 404 returned error can't find the container with id e99617ee2acfb31b9bb76392e4f7a01e30887a50e6dd2fc69bd57116d7337397 Oct 31 21:35:56.257388 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:56.257347 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz"] Oct 31 21:35:56.607285 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:56.607239 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" event={"ID":"3c6373a6-6f55-4004-a7b1-66ada946c9be","Type":"ContainerStarted","Data":"34f228c58be3eaa376b80c230963d4e0d4997027dde660c35ada35545bf607e7"} Oct 31 21:35:56.607484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:56.607362 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" event={"ID":"3c6373a6-6f55-4004-a7b1-66ada946c9be","Type":"ContainerStarted","Data":"e99617ee2acfb31b9bb76392e4f7a01e30887a50e6dd2fc69bd57116d7337397"} Oct 31 21:35:56.646063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:56.646003 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" podStartSLOduration=1.6459877600000001 podStartE2EDuration="1.64598776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:35:56.644458901 +0000 UTC m=+1203.337416305" watchObservedRunningTime="2026-10-31 21:35:56.64598776 +0000 UTC m=+1203.338945152" Oct 31 21:35:57.620026 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:57.619979 5188 generic.go:356] "Generic (PLEG): container finished" podID="3c6373a6-6f55-4004-a7b1-66ada946c9be" containerID="34f228c58be3eaa376b80c230963d4e0d4997027dde660c35ada35545bf607e7" exitCode=0 Oct 31 21:35:57.620535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:57.620053 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" event={"ID":"3c6373a6-6f55-4004-a7b1-66ada946c9be","Type":"ContainerDied","Data":"34f228c58be3eaa376b80c230963d4e0d4997027dde660c35ada35545bf607e7"} Oct 31 21:35:58.801514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.801488 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:58.890583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.890479 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-xks-ca\") pod \"3c6373a6-6f55-4004-a7b1-66ada946c9be\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " Oct 31 21:35:58.890583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.890543 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-openshift-ca\") pod \"3c6373a6-6f55-4004-a7b1-66ada946c9be\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " Oct 31 21:35:58.891413 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.890596 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6373a6-6f55-4004-a7b1-66ada946c9be-kube-api-access-lqpw5\" (UniqueName: \"kubernetes.io/projected/3c6373a6-6f55-4004-a7b1-66ada946c9be-kube-api-access-lqpw5\") pod \"3c6373a6-6f55-4004-a7b1-66ada946c9be\" (UID: \"3c6373a6-6f55-4004-a7b1-66ada946c9be\") " Oct 31 21:35:58.891538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.891289 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-openshift-ca" pod "3c6373a6-6f55-4004-a7b1-66ada946c9be" (UID: "3c6373a6-6f55-4004-a7b1-66ada946c9be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:35:58.892291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.892265 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:58.894113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.893372 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6373a6-6f55-4004-a7b1-66ada946c9be-kube-api-access-lqpw5" pod "3c6373a6-6f55-4004-a7b1-66ada946c9be" (UID: "3c6373a6-6f55-4004-a7b1-66ada946c9be"). InnerVolumeSpecName "kube-api-access-lqpw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:58.895127 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.895049 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-xks-ca" pod "3c6373a6-6f55-4004-a7b1-66ada946c9be" (UID: "3c6373a6-6f55-4004-a7b1-66ada946c9be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:35:58.994160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.994124 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c6373a6-6f55-4004-a7b1-66ada946c9be-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:58.994160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:58.994156 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqpw5\" (UniqueName: \"kubernetes.io/projected/3c6373a6-6f55-4004-a7b1-66ada946c9be-kube-api-access-lqpw5\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:35:59.638969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:59.638934 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" Oct 31 21:35:59.638969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:59.638953 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz" event={"ID":"3c6373a6-6f55-4004-a7b1-66ada946c9be","Type":"ContainerDied","Data":"e99617ee2acfb31b9bb76392e4f7a01e30887a50e6dd2fc69bd57116d7337397"} Oct 31 21:35:59.639199 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:35:59.639014 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99617ee2acfb31b9bb76392e4f7a01e30887a50e6dd2fc69bd57116d7337397" Oct 31 21:36:00.316945 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.316916 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:36:00.432612 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.432573 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-credential-socket\") pod \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " Oct 31 21:36:00.432819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.432633 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-podinfo\") pod \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " Oct 31 21:36:00.432819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.432720 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-token\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-token\") pod \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " Oct 31 21:36:00.432819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.432752 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istiod-ca-cert\") pod \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " Oct 31 21:36:00.432990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.432860 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-certs\") pod \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " Oct 31 21:36:00.432990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.432898 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-socket\") pod \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " Oct 31 21:36:00.432990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.432926 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-credential-socket" pod "ae718dc5-7881-4f3f-a98d-a517cefd0ab4" (UID: "ae718dc5-7881-4f3f-a98d-a517cefd0ab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:36:00.433153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.433026 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-data\") pod \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " Oct 31 21:36:00.433153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.433122 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-envoy\") pod \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " Oct 31 21:36:00.433262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.433148 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-certs" pod "ae718dc5-7881-4f3f-a98d-a517cefd0ab4" (UID: "ae718dc5-7881-4f3f-a98d-a517cefd0ab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:36:00.433262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.433202 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-kube-api-access-hcxlz\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-kube-api-access-hcxlz\") pod \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\" (UID: \"ae718dc5-7881-4f3f-a98d-a517cefd0ab4\") " Oct 31 21:36:00.433380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.433266 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-socket" pod "ae718dc5-7881-4f3f-a98d-a517cefd0ab4" (UID: "ae718dc5-7881-4f3f-a98d-a517cefd0ab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:36:00.433380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.433196 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istiod-ca-cert" pod "ae718dc5-7881-4f3f-a98d-a517cefd0ab4" (UID: "ae718dc5-7881-4f3f-a98d-a517cefd0ab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:36:00.433479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.433377 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-data" pod "ae718dc5-7881-4f3f-a98d-a517cefd0ab4" (UID: "ae718dc5-7881-4f3f-a98d-a517cefd0ab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:36:00.433997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.433972 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:00.434110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.434000 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:00.434110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.434016 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:00.434110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.434028 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:00.434110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.434042 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:00.440713 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.440681 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-envoy" pod "ae718dc5-7881-4f3f-a98d-a517cefd0ab4" (UID: "ae718dc5-7881-4f3f-a98d-a517cefd0ab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:36:00.440815 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.440715 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-podinfo" pod "ae718dc5-7881-4f3f-a98d-a517cefd0ab4" (UID: "ae718dc5-7881-4f3f-a98d-a517cefd0ab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:36:00.440815 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.440742 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-kube-api-access-hcxlz" pod "ae718dc5-7881-4f3f-a98d-a517cefd0ab4" (UID: "ae718dc5-7881-4f3f-a98d-a517cefd0ab4"). InnerVolumeSpecName "kube-api-access-hcxlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:00.440815 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.440769 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-token" pod "ae718dc5-7881-4f3f-a98d-a517cefd0ab4" (UID: "ae718dc5-7881-4f3f-a98d-a517cefd0ab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:00.536019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.535932 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:00.536019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.535965 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcxlz\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-kube-api-access-hcxlz\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:00.536019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.535978 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:00.536019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.535987 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae718dc5-7881-4f3f-a98d-a517cefd0ab4-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:00.654378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.653578 5188 generic.go:356] "Generic (PLEG): container finished" podID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerID="439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d" exitCode=0 Oct 31 21:36:00.654378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.653660 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" event={"ID":"ae718dc5-7881-4f3f-a98d-a517cefd0ab4","Type":"ContainerDied","Data":"439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d"} Oct 31 21:36:00.654378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.653715 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" Oct 31 21:36:00.654378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.653728 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf" event={"ID":"ae718dc5-7881-4f3f-a98d-a517cefd0ab4","Type":"ContainerDied","Data":"c29ee9d0177e1ae1d29f5c83177fd118b310b7765145826f0ac88840aacd4b15"} Oct 31 21:36:00.654378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.653740 5188 scope.go:122] "RemoveContainer" containerID="439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d" Oct 31 21:36:00.684013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.683977 5188 scope.go:122] "RemoveContainer" containerID="827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d" Oct 31 21:36:00.698273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.698233 5188 scope.go:122] "RemoveContainer" containerID="439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d" Oct 31 21:36:00.698884 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:36:00.698845 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d\": container with ID starting with 439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d not found: ID does not exist" containerID="439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d" Oct 31 21:36:00.698990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.698902 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d"} err="failed to get container status \"439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d\": rpc error: code = NotFound desc = could not find container \"439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d\": container with ID starting with 439a731489218a8ea4b03adb9be4843a508951591cbb8ea9d87e76adc6c9b16d not found: ID does not exist" Oct 31 21:36:00.698990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.698939 5188 scope.go:122] "RemoveContainer" containerID="827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d" Oct 31 21:36:00.699363 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:36:00.699302 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d\": container with ID starting with 827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d not found: ID does not exist" containerID="827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d" Oct 31 21:36:00.699363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.699350 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d"} err="failed to get container status \"827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d\": rpc error: code = NotFound desc = could not find container \"827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d\": container with ID starting with 827457090d94eaf51678fa2b42d11c56851e4fff5960b51291b206908858139d not found: ID does not exist" Oct 31 21:36:00.724523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.724491 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf"] Oct 31 21:36:00.736126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:00.736086 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7ac3c8-openshift-default-845c59dc47-wckpf"] Oct 31 21:36:01.033258 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:01.033214 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf"] Oct 31 21:36:01.040600 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:01.040556 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-912f1738-fgnhf"] Oct 31 21:36:01.475809 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:01.475766 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" path="/var/lib/kubelet/pods/ae718dc5-7881-4f3f-a98d-a517cefd0ab4/volumes" Oct 31 21:36:01.476379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:01.476359 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c1703c-89ee-4fda-9b11-c705c3c2ad0b" path="/var/lib/kubelet/pods/b0c1703c-89ee-4fda-9b11-c705c3c2ad0b/volumes" Oct 31 21:36:02.033042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:02.032999 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz"] Oct 31 21:36:02.042154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:02.042121 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f40e774-lpqkz"] Oct 31 21:36:02.539573 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:36:02.539537 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:36:03.482484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:03.482448 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371aa73a-ad00-45eb-8216-d8f9116e4577" path="/var/lib/kubelet/pods/371aa73a-ad00-45eb-8216-d8f9116e4577/volumes" Oct 31 21:36:03.904697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:03.904599 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n"] Oct 31 21:36:03.910998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:03.910964 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-59z7n"] Oct 31 21:36:04.526495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:04.526466 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924"] Oct 31 21:36:04.534355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:04.534305 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-371531-g5924"] Oct 31 21:36:05.036145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:05.036108 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4"] Oct 31 21:36:05.048076 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:05.048043 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e9dadda-mvbm4"] Oct 31 21:36:05.475650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:05.475610 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17258272-6b47-43d4-929b-62e74e9b3ce4" path="/var/lib/kubelet/pods/17258272-6b47-43d4-929b-62e74e9b3ce4/volumes" Oct 31 21:36:05.476215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:05.476197 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69ca93b-9bf1-441f-9df0-2e03823fdb2a" path="/var/lib/kubelet/pods/c69ca93b-9bf1-441f-9df0-2e03823fdb2a/volumes" Oct 31 21:36:05.476670 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:05.476656 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa31abc-9895-4597-9603-90863762aae0" path="/var/lib/kubelet/pods/daa31abc-9895-4597-9603-90863762aae0/volumes" Oct 31 21:36:17.038931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:17.038871 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p"] Oct 31 21:36:17.048313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:17.048271 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9128dcf4-xfn9p"] Oct 31 21:36:17.476813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:17.476775 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8338233-eadd-47de-806c-8bae1b0617cf" path="/var/lib/kubelet/pods/d8338233-eadd-47de-806c-8bae1b0617cf/volumes" Oct 31 21:36:18.037676 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:18.037639 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7"] Oct 31 21:36:18.052701 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:18.052656 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34a3de07-w5gd7"] Oct 31 21:36:19.475745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:19.475706 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d360b3a3-537c-4fa1-90c7-72ad0d76d6f5" path="/var/lib/kubelet/pods/d360b3a3-537c-4fa1-90c7-72ad0d76d6f5/volumes" Oct 31 21:36:21.710877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.710841 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn"] Oct 31 21:36:21.711566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.711541 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" Oct 31 21:36:21.711632 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.711566 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" Oct 31 21:36:21.711632 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.711576 5188 cpu_manager.go:414] "containerMap: removing container" podUID="3c6373a6-6f55-4004-a7b1-66ada946c9be" containerName="revoke-keys" Oct 31 21:36:21.711632 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.711582 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6373a6-6f55-4004-a7b1-66ada946c9be" containerName="revoke-keys" Oct 31 21:36:21.711632 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.711593 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" Oct 31 21:36:21.711632 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.711598 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" Oct 31 21:36:21.711786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.711726 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" Oct 31 21:36:21.711786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.711735 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6373a6-6f55-4004-a7b1-66ada946c9be" containerName="revoke-keys" Oct 31 21:36:21.711786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.711742 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae718dc5-7881-4f3f-a98d-a517cefd0ab4" containerName="istio-proxy" Oct 31 21:36:21.716894 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.716870 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:21.729412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.729312 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:36:21.729589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.729395 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:36:21.729686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.729607 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:36:21.729944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.729733 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:36:21.735467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.735435 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn"] Oct 31 21:36:21.845866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.845826 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:21.846062 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.845890 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgn5h\" (UniqueName: \"kubernetes.io/projected/2ebe95f0-7191-4e09-bb55-5a144b8b2613-kube-api-access-mgn5h\") pod \"maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:21.846062 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.846034 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:21.948106 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.948048 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:21.948336 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.948157 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgn5h\" (UniqueName: \"kubernetes.io/projected/2ebe95f0-7191-4e09-bb55-5a144b8b2613-kube-api-access-mgn5h\") pod \"maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:21.948336 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.948259 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:21.948998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.948970 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:21.950929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.950894 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:21.970038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:21.969971 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgn5h\" (UniqueName: \"kubernetes.io/projected/2ebe95f0-7191-4e09-bb55-5a144b8b2613-kube-api-access-mgn5h\") pod \"maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:22.032085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:22.032047 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:22.348512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:22.348416 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn"] Oct 31 21:36:22.352286 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:36:22.352254 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ebe95f0_7191_4e09_bb55_5a144b8b2613.slice/crio-6723c4a45f40d4326e46fee79bad79cdecf65a4b5edc5629b875ea8a112259d9 WatchSource:0}: Error finding container 6723c4a45f40d4326e46fee79bad79cdecf65a4b5edc5629b875ea8a112259d9: Status 404 returned error can't find the container with id 6723c4a45f40d4326e46fee79bad79cdecf65a4b5edc5629b875ea8a112259d9 Oct 31 21:36:22.998580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:22.998541 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" event={"ID":"2ebe95f0-7191-4e09-bb55-5a144b8b2613","Type":"ContainerStarted","Data":"977e597e80f9f064032b08ed49147191224c8f30c0f085b2cf112ffe381f0b9f"} Oct 31 21:36:22.999014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:22.998601 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" event={"ID":"2ebe95f0-7191-4e09-bb55-5a144b8b2613","Type":"ContainerStarted","Data":"6723c4a45f40d4326e46fee79bad79cdecf65a4b5edc5629b875ea8a112259d9"} Oct 31 21:36:23.047456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:23.047311 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" podStartSLOduration=2.047283934 podStartE2EDuration="2.047283934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:36:23.03422166 +0000 UTC m=+1229.727179053" watchObservedRunningTime="2026-10-31 21:36:23.047283934 +0000 UTC m=+1229.740241329" Oct 31 21:36:24.009868 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:24.009832 5188 generic.go:356] "Generic (PLEG): container finished" podID="2ebe95f0-7191-4e09-bb55-5a144b8b2613" containerID="977e597e80f9f064032b08ed49147191224c8f30c0f085b2cf112ffe381f0b9f" exitCode=0 Oct 31 21:36:24.010298 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:24.009919 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" event={"ID":"2ebe95f0-7191-4e09-bb55-5a144b8b2613","Type":"ContainerDied","Data":"977e597e80f9f064032b08ed49147191224c8f30c0f085b2cf112ffe381f0b9f"} Oct 31 21:36:25.193689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.193659 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:25.312890 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.312788 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-openshift-ca\") pod \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " Oct 31 21:36:25.313064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.312890 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-xks-ca\") pod \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " Oct 31 21:36:25.313064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.312943 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ebe95f0-7191-4e09-bb55-5a144b8b2613-kube-api-access-mgn5h\" (UniqueName: \"kubernetes.io/projected/2ebe95f0-7191-4e09-bb55-5a144b8b2613-kube-api-access-mgn5h\") pod \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\" (UID: \"2ebe95f0-7191-4e09-bb55-5a144b8b2613\") " Oct 31 21:36:25.313309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.313281 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-openshift-ca" pod "2ebe95f0-7191-4e09-bb55-5a144b8b2613" (UID: "2ebe95f0-7191-4e09-bb55-5a144b8b2613"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:36:25.313557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.313535 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:25.315429 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.315386 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-xks-ca" pod "2ebe95f0-7191-4e09-bb55-5a144b8b2613" (UID: "2ebe95f0-7191-4e09-bb55-5a144b8b2613"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:36:25.315709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.315680 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ebe95f0-7191-4e09-bb55-5a144b8b2613-kube-api-access-mgn5h" pod "2ebe95f0-7191-4e09-bb55-5a144b8b2613" (UID: "2ebe95f0-7191-4e09-bb55-5a144b8b2613"). InnerVolumeSpecName "kube-api-access-mgn5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:25.415126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.415086 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ebe95f0-7191-4e09-bb55-5a144b8b2613-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:25.415126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:25.415120 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgn5h\" (UniqueName: \"kubernetes.io/projected/2ebe95f0-7191-4e09-bb55-5a144b8b2613-kube-api-access-mgn5h\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:26.031704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:26.031668 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" Oct 31 21:36:26.031908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:26.031700 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn" event={"ID":"2ebe95f0-7191-4e09-bb55-5a144b8b2613","Type":"ContainerDied","Data":"6723c4a45f40d4326e46fee79bad79cdecf65a4b5edc5629b875ea8a112259d9"} Oct 31 21:36:26.031908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:26.031764 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6723c4a45f40d4326e46fee79bad79cdecf65a4b5edc5629b875ea8a112259d9" Oct 31 21:36:28.371140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:28.371100 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz"] Oct 31 21:36:28.381241 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:28.381203 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-248b29-stxvz"] Oct 31 21:36:29.033579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:29.033540 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2"] Oct 31 21:36:29.044808 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:29.044772 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add1ab35-lgxs2"] Oct 31 21:36:29.475907 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:29.475863 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6373a6-6f55-4004-a7b1-66ada946c9be" path="/var/lib/kubelet/pods/3c6373a6-6f55-4004-a7b1-66ada946c9be/volumes" Oct 31 21:36:29.476640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:29.476617 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd" path="/var/lib/kubelet/pods/f5adc2f7-42fc-4615-9732-cc7b9c9d8dcd/volumes" Oct 31 21:36:32.561313 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:36:32.561277 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:36:36.032870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.032832 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4"] Oct 31 21:36:36.043466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.043425 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9cee038-t64b4"] Oct 31 21:36:36.667613 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.667569 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp"] Oct 31 21:36:36.685112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.681073 5188 cpu_manager.go:414] "containerMap: removing container" podUID="2ebe95f0-7191-4e09-bb55-5a144b8b2613" containerName="revoke-keys" Oct 31 21:36:36.685112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.681106 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ebe95f0-7191-4e09-bb55-5a144b8b2613" containerName="revoke-keys" Oct 31 21:36:36.685112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.681252 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ebe95f0-7191-4e09-bb55-5a144b8b2613" containerName="revoke-keys" Oct 31 21:36:36.686777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.686737 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp"] Oct 31 21:36:36.687023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.686834 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:36.709649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.709620 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:36:36.709832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.709665 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:36:36.709928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.709694 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:36:36.709928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.709752 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:36:36.877757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.877719 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8424d9d-cjrdp\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:36.877982 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.877771 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9nn\" (UniqueName: \"kubernetes.io/projected/8ebdccf2-7fce-4054-a600-e50208152f4d-kube-api-access-cb9nn\") pod \"maas-api-revoke-sub-a8424d9d-cjrdp\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:36.877982 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.877926 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8424d9d-cjrdp\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:36.980038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.979942 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8424d9d-cjrdp\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:36.980352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.980304 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8424d9d-cjrdp\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:36.980414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.980379 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9nn\" (UniqueName: \"kubernetes.io/projected/8ebdccf2-7fce-4054-a600-e50208152f4d-kube-api-access-cb9nn\") pod \"maas-api-revoke-sub-a8424d9d-cjrdp\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:36.980818 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.980791 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8424d9d-cjrdp\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:36.982890 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:36.982862 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8424d9d-cjrdp\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:37.002799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:37.002763 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9nn\" (UniqueName: \"kubernetes.io/projected/8ebdccf2-7fce-4054-a600-e50208152f4d-kube-api-access-cb9nn\") pod \"maas-api-revoke-sub-a8424d9d-cjrdp\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:37.013077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:37.013029 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:37.333141 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:36:37.333104 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebdccf2_7fce_4054_a600_e50208152f4d.slice/crio-f93be10ecf12fcd99645dac16ef99f50734f441e5617a9af394692aa5648b4bd WatchSource:0}: Error finding container f93be10ecf12fcd99645dac16ef99f50734f441e5617a9af394692aa5648b4bd: Status 404 returned error can't find the container with id f93be10ecf12fcd99645dac16ef99f50734f441e5617a9af394692aa5648b4bd Oct 31 21:36:37.333518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:37.333462 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp"] Oct 31 21:36:37.542356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:37.534656 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0ec435-3c2a-442f-b2f5-93994c9d5e73" path="/var/lib/kubelet/pods/6d0ec435-3c2a-442f-b2f5-93994c9d5e73/volumes" Oct 31 21:36:38.226715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:38.226674 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" event={"ID":"8ebdccf2-7fce-4054-a600-e50208152f4d","Type":"ContainerStarted","Data":"aba81d54be71901e2a1c8e9cfa82a4b7c80168c8daca64428cda8d378b27e056"} Oct 31 21:36:38.226929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:38.226737 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" event={"ID":"8ebdccf2-7fce-4054-a600-e50208152f4d","Type":"ContainerStarted","Data":"f93be10ecf12fcd99645dac16ef99f50734f441e5617a9af394692aa5648b4bd"} Oct 31 21:36:38.265989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:38.265929 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" podStartSLOduration=2.265912329 podStartE2EDuration="2.265912329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:36:38.263298654 +0000 UTC m=+1244.956256046" watchObservedRunningTime="2026-10-31 21:36:38.265912329 +0000 UTC m=+1244.958869721" Oct 31 21:36:39.238759 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:39.238717 5188 generic.go:356] "Generic (PLEG): container finished" podID="8ebdccf2-7fce-4054-a600-e50208152f4d" containerID="aba81d54be71901e2a1c8e9cfa82a4b7c80168c8daca64428cda8d378b27e056" exitCode=0 Oct 31 21:36:39.239193 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:39.238790 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" event={"ID":"8ebdccf2-7fce-4054-a600-e50208152f4d","Type":"ContainerDied","Data":"aba81d54be71901e2a1c8e9cfa82a4b7c80168c8daca64428cda8d378b27e056"} Oct 31 21:36:40.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.033510 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv"] Oct 31 21:36:40.044714 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.044673 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f91a1c-lxnnv"] Oct 31 21:36:40.429152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.429125 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:40.565272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.565178 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-openshift-ca\") pod \"8ebdccf2-7fce-4054-a600-e50208152f4d\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " Oct 31 21:36:40.565272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.565256 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ebdccf2-7fce-4054-a600-e50208152f4d-kube-api-access-cb9nn\" (UniqueName: \"kubernetes.io/projected/8ebdccf2-7fce-4054-a600-e50208152f4d-kube-api-access-cb9nn\") pod \"8ebdccf2-7fce-4054-a600-e50208152f4d\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " Oct 31 21:36:40.565518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.565292 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-xks-ca\") pod \"8ebdccf2-7fce-4054-a600-e50208152f4d\" (UID: \"8ebdccf2-7fce-4054-a600-e50208152f4d\") " Oct 31 21:36:40.565647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.565618 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-openshift-ca" pod "8ebdccf2-7fce-4054-a600-e50208152f4d" (UID: "8ebdccf2-7fce-4054-a600-e50208152f4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:36:40.565987 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.565968 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:40.567816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.567790 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-xks-ca" pod "8ebdccf2-7fce-4054-a600-e50208152f4d" (UID: "8ebdccf2-7fce-4054-a600-e50208152f4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:36:40.567932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.567832 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebdccf2-7fce-4054-a600-e50208152f4d-kube-api-access-cb9nn" pod "8ebdccf2-7fce-4054-a600-e50208152f4d" (UID: "8ebdccf2-7fce-4054-a600-e50208152f4d"). InnerVolumeSpecName "kube-api-access-cb9nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:40.668897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.668860 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb9nn\" (UniqueName: \"kubernetes.io/projected/8ebdccf2-7fce-4054-a600-e50208152f4d-kube-api-access-cb9nn\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:40.668897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:40.668895 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebdccf2-7fce-4054-a600-e50208152f4d-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:41.324345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:41.324178 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" event={"ID":"8ebdccf2-7fce-4054-a600-e50208152f4d","Type":"ContainerDied","Data":"f93be10ecf12fcd99645dac16ef99f50734f441e5617a9af394692aa5648b4bd"} Oct 31 21:36:41.324345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:41.324250 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93be10ecf12fcd99645dac16ef99f50734f441e5617a9af394692aa5648b4bd" Oct 31 21:36:41.324345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:41.324337 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp" Oct 31 21:36:41.492471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:41.492430 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86b8fab-cad6-4283-956f-c17e93c928c3" path="/var/lib/kubelet/pods/e86b8fab-cad6-4283-956f-c17e93c928c3/volumes" Oct 31 21:36:44.035126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:44.035085 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk"] Oct 31 21:36:44.044106 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:44.044071 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92704f8a-kkkvk"] Oct 31 21:36:45.478615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.478576 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad0f03a-910a-4772-9c6a-9c7624c81da4" path="/var/lib/kubelet/pods/bad0f03a-910a-4772-9c6a-9c7624c81da4/volumes" Oct 31 21:36:45.682561 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.682515 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz"] Oct 31 21:36:45.683671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.683636 5188 cpu_manager.go:414] "containerMap: removing container" podUID="8ebdccf2-7fce-4054-a600-e50208152f4d" containerName="revoke-keys" Oct 31 21:36:45.683671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.683665 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ebdccf2-7fce-4054-a600-e50208152f4d" containerName="revoke-keys" Oct 31 21:36:45.683931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.683915 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ebdccf2-7fce-4054-a600-e50208152f4d" containerName="revoke-keys" Oct 31 21:36:45.690132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.690102 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:45.706271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.706234 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz"] Oct 31 21:36:45.707944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.707916 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:36:45.708109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.707965 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:36:45.708109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.708044 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:36:45.710943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.710900 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-wgrgz\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:45.711119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.711098 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-wgrgz\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:45.711318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.711295 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdms\" (UniqueName: \"kubernetes.io/projected/6d63cbad-61eb-4678-abe6-02ee5810d710-kube-api-access-vrdms\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-wgrgz\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:45.712295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.712275 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:36:45.812880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.812786 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-wgrgz\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:45.813067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.812904 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-wgrgz\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:45.813141 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.813063 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdms\" (UniqueName: \"kubernetes.io/projected/6d63cbad-61eb-4678-abe6-02ee5810d710-kube-api-access-vrdms\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-wgrgz\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:45.813570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.813543 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-wgrgz\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:45.815629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.815597 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-wgrgz\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:45.834399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:45.834346 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdms\" (UniqueName: \"kubernetes.io/projected/6d63cbad-61eb-4678-abe6-02ee5810d710-kube-api-access-vrdms\") pod \"maas-api-revoke-keys-e2e-mt-084d807f-wgrgz\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:46.010226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:46.010182 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:46.333604 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:36:46.333562 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d63cbad_61eb_4678_abe6_02ee5810d710.slice/crio-39085bb021a0a29627d2b45aa45b47aa34193b88221be5556d9b9dbcfd0ec1e1 WatchSource:0}: Error finding container 39085bb021a0a29627d2b45aa45b47aa34193b88221be5556d9b9dbcfd0ec1e1: Status 404 returned error can't find the container with id 39085bb021a0a29627d2b45aa45b47aa34193b88221be5556d9b9dbcfd0ec1e1 Oct 31 21:36:46.336121 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:46.336087 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz"] Oct 31 21:36:46.374673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:46.374637 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" event={"ID":"6d63cbad-61eb-4678-abe6-02ee5810d710","Type":"ContainerStarted","Data":"39085bb021a0a29627d2b45aa45b47aa34193b88221be5556d9b9dbcfd0ec1e1"} Oct 31 21:36:47.387951 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:47.387904 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" event={"ID":"6d63cbad-61eb-4678-abe6-02ee5810d710","Type":"ContainerStarted","Data":"83f09aa4d8a4454b4b4be7b38beb578a734ddea9bcffb09fe4803121580ac66c"} Oct 31 21:36:47.428040 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:47.427964 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" podStartSLOduration=2.427941247 podStartE2EDuration="2.427941247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:36:47.423293419 +0000 UTC m=+1254.116250813" watchObservedRunningTime="2026-10-31 21:36:47.427941247 +0000 UTC m=+1254.120898643" Oct 31 21:36:48.400885 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:48.400846 5188 generic.go:356] "Generic (PLEG): container finished" podID="6d63cbad-61eb-4678-abe6-02ee5810d710" containerID="83f09aa4d8a4454b4b4be7b38beb578a734ddea9bcffb09fe4803121580ac66c" exitCode=0 Oct 31 21:36:48.401484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:48.400930 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" event={"ID":"6d63cbad-61eb-4678-abe6-02ee5810d710","Type":"ContainerDied","Data":"83f09aa4d8a4454b4b4be7b38beb578a734ddea9bcffb09fe4803121580ac66c"} Oct 31 21:36:49.581029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.580997 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:49.685439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.685317 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-xks-ca\") pod \"6d63cbad-61eb-4678-abe6-02ee5810d710\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " Oct 31 21:36:49.685967 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.685936 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-openshift-ca\") pod \"6d63cbad-61eb-4678-abe6-02ee5810d710\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " Oct 31 21:36:49.686111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.686015 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d63cbad-61eb-4678-abe6-02ee5810d710-kube-api-access-vrdms\" (UniqueName: \"kubernetes.io/projected/6d63cbad-61eb-4678-abe6-02ee5810d710-kube-api-access-vrdms\") pod \"6d63cbad-61eb-4678-abe6-02ee5810d710\" (UID: \"6d63cbad-61eb-4678-abe6-02ee5810d710\") " Oct 31 21:36:49.686387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.686355 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-openshift-ca" pod "6d63cbad-61eb-4678-abe6-02ee5810d710" (UID: "6d63cbad-61eb-4678-abe6-02ee5810d710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:36:49.686788 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.686763 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:49.687706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.687682 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-xks-ca" pod "6d63cbad-61eb-4678-abe6-02ee5810d710" (UID: "6d63cbad-61eb-4678-abe6-02ee5810d710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:36:49.690089 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.690064 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d63cbad-61eb-4678-abe6-02ee5810d710-kube-api-access-vrdms" pod "6d63cbad-61eb-4678-abe6-02ee5810d710" (UID: "6d63cbad-61eb-4678-abe6-02ee5810d710"). InnerVolumeSpecName "kube-api-access-vrdms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:49.788207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.788171 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrdms\" (UniqueName: \"kubernetes.io/projected/6d63cbad-61eb-4678-abe6-02ee5810d710-kube-api-access-vrdms\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:49.788207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:49.788202 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d63cbad-61eb-4678-abe6-02ee5810d710-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:36:50.422554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:50.422520 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" Oct 31 21:36:50.422718 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:50.422513 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz" event={"ID":"6d63cbad-61eb-4678-abe6-02ee5810d710","Type":"ContainerDied","Data":"39085bb021a0a29627d2b45aa45b47aa34193b88221be5556d9b9dbcfd0ec1e1"} Oct 31 21:36:50.422718 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:50.422659 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39085bb021a0a29627d2b45aa45b47aa34193b88221be5556d9b9dbcfd0ec1e1" Oct 31 21:36:54.652716 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:54.652682 5188 scope.go:122] "RemoveContainer" containerID="2d186d0683cfe09c2023d4698937085838ade80bfa6f1247484b05187faa43d6" Oct 31 21:36:56.271452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:56.271399 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc"] Oct 31 21:36:56.314895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:56.314429 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:36:56.418087 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:56.418030 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:36:56.490780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:56.488212 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6593169-vs8hc"] Oct 31 21:36:56.564398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:56.560516 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 21:36:57.322347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.320415 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Oct 31 21:36:57.322347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.321709 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:36:57.353585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.353534 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 31 21:36:57.482349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.481611 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:36:57.516223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.514954 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:36:57.580559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.580444 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:36:57.832409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.804907 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd1ac49-8c79-4ac3-bc18-f385b79299d9" path="/var/lib/kubelet/pods/2cd1ac49-8c79-4ac3-bc18-f385b79299d9/volumes" Oct 31 21:36:57.939350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.913161 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:36:57.939350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.915608 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 21:36:57.939668 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.939610 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn"] Oct 31 21:36:57.939741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.939693 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 21:36:57.940437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:57.940407 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 21:36:58.027653 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:58.027615 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08ddf0-xpwdn"] Oct 31 21:36:59.475473 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:36:59.475436 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ebe95f0-7191-4e09-bb55-5a144b8b2613" path="/var/lib/kubelet/pods/2ebe95f0-7191-4e09-bb55-5a144b8b2613/volumes" Oct 31 21:37:02.034896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:02.034851 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s"] Oct 31 21:37:02.045005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:02.044964 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-999edc59-bks2s"] Oct 31 21:37:02.548934 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:37:02.548893 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:37:03.478574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:03.478526 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75599e2d-53bd-4bd8-bc55-85ececa49702" path="/var/lib/kubelet/pods/75599e2d-53bd-4bd8-bc55-85ececa49702/volumes" Oct 31 21:37:05.792374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.790717 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x"] Oct 31 21:37:05.792374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.791457 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6d63cbad-61eb-4678-abe6-02ee5810d710" containerName="revoke-keys" Oct 31 21:37:05.792374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.791473 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d63cbad-61eb-4678-abe6-02ee5810d710" containerName="revoke-keys" Oct 31 21:37:05.792374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.791659 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d63cbad-61eb-4678-abe6-02ee5810d710" containerName="revoke-keys" Oct 31 21:37:05.797821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.797789 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:05.818238 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.818202 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x"] Oct 31 21:37:05.819963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.819927 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:37:05.820213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.820076 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:37:05.820276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.820230 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:37:05.821549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.821504 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:37:05.909054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.909000 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksj8r\" (UniqueName: \"kubernetes.io/projected/9649d783-a478-4ee8-8de0-97573500d2de-kube-api-access-ksj8r\") pod \"maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:05.909236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.909128 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:05.909302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:05.909286 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:06.011312 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.011268 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:06.011583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.011558 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksj8r\" (UniqueName: \"kubernetes.io/projected/9649d783-a478-4ee8-8de0-97573500d2de-kube-api-access-ksj8r\") pod \"maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:06.011672 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.011630 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:06.012166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.012146 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:06.014209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.014190 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:06.034092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.034052 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m"] Oct 31 21:37:06.035809 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.035776 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksj8r\" (UniqueName: \"kubernetes.io/projected/9649d783-a478-4ee8-8de0-97573500d2de-kube-api-access-ksj8r\") pod \"maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:06.039012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.038986 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c6edead-95r5m"] Oct 31 21:37:06.122068 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.122034 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:06.434508 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:37:06.434465 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9649d783_a478_4ee8_8de0_97573500d2de.slice/crio-d01c10f1e6a6f23be38813740c1335cb64a79bb94dc3dbc0601f8c2a0e696441 WatchSource:0}: Error finding container d01c10f1e6a6f23be38813740c1335cb64a79bb94dc3dbc0601f8c2a0e696441: Status 404 returned error can't find the container with id d01c10f1e6a6f23be38813740c1335cb64a79bb94dc3dbc0601f8c2a0e696441 Oct 31 21:37:06.435025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:06.434999 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x"] Oct 31 21:37:07.316395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:07.316357 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" event={"ID":"9649d783-a478-4ee8-8de0-97573500d2de","Type":"ContainerStarted","Data":"cd8cbd44fe059ac5743b0c61c758a73d286e226c84d1ffaf405f3b0b36bac325"} Oct 31 21:37:07.316897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:07.316421 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" event={"ID":"9649d783-a478-4ee8-8de0-97573500d2de","Type":"ContainerStarted","Data":"d01c10f1e6a6f23be38813740c1335cb64a79bb94dc3dbc0601f8c2a0e696441"} Oct 31 21:37:07.356917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:07.356864 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" podStartSLOduration=2.356850787 podStartE2EDuration="2.356850787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:37:07.351644794 +0000 UTC m=+1274.044602187" watchObservedRunningTime="2026-10-31 21:37:07.356850787 +0000 UTC m=+1274.049808180" Oct 31 21:37:07.481769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:07.481726 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752a24f7-0fe5-4ffe-9a8e-c590ed2500f5" path="/var/lib/kubelet/pods/752a24f7-0fe5-4ffe-9a8e-c590ed2500f5/volumes" Oct 31 21:37:08.327524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:08.327487 5188 generic.go:356] "Generic (PLEG): container finished" podID="9649d783-a478-4ee8-8de0-97573500d2de" containerID="cd8cbd44fe059ac5743b0c61c758a73d286e226c84d1ffaf405f3b0b36bac325" exitCode=0 Oct 31 21:37:08.328013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:08.327574 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" event={"ID":"9649d783-a478-4ee8-8de0-97573500d2de","Type":"ContainerDied","Data":"cd8cbd44fe059ac5743b0c61c758a73d286e226c84d1ffaf405f3b0b36bac325"} Oct 31 21:37:09.542910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.541400 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz"] Oct 31 21:37:09.575912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.565792 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-084d807f-wgrgz"] Oct 31 21:37:09.622435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.622409 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:09.741315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.739315 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-openshift-ca\") pod \"9649d783-a478-4ee8-8de0-97573500d2de\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " Oct 31 21:37:09.741315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.739385 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9649d783-a478-4ee8-8de0-97573500d2de-kube-api-access-ksj8r\" (UniqueName: \"kubernetes.io/projected/9649d783-a478-4ee8-8de0-97573500d2de-kube-api-access-ksj8r\") pod \"9649d783-a478-4ee8-8de0-97573500d2de\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " Oct 31 21:37:09.741315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.739454 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-xks-ca\") pod \"9649d783-a478-4ee8-8de0-97573500d2de\" (UID: \"9649d783-a478-4ee8-8de0-97573500d2de\") " Oct 31 21:37:09.741315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.740020 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-openshift-ca" pod "9649d783-a478-4ee8-8de0-97573500d2de" (UID: "9649d783-a478-4ee8-8de0-97573500d2de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:37:09.741315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.740194 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:37:09.754522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.749282 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-xks-ca" pod "9649d783-a478-4ee8-8de0-97573500d2de" (UID: "9649d783-a478-4ee8-8de0-97573500d2de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:37:09.764194 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.764140 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9649d783-a478-4ee8-8de0-97573500d2de-kube-api-access-ksj8r" pod "9649d783-a478-4ee8-8de0-97573500d2de" (UID: "9649d783-a478-4ee8-8de0-97573500d2de"). InnerVolumeSpecName "kube-api-access-ksj8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:37:09.841852 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.841817 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksj8r\" (UniqueName: \"kubernetes.io/projected/9649d783-a478-4ee8-8de0-97573500d2de-kube-api-access-ksj8r\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:37:09.841852 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:09.841850 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9649d783-a478-4ee8-8de0-97573500d2de-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:37:10.404505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:10.404466 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" Oct 31 21:37:10.404505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:10.404498 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x" event={"ID":"9649d783-a478-4ee8-8de0-97573500d2de","Type":"ContainerDied","Data":"d01c10f1e6a6f23be38813740c1335cb64a79bb94dc3dbc0601f8c2a0e696441"} Oct 31 21:37:10.404744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:10.404551 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d01c10f1e6a6f23be38813740c1335cb64a79bb94dc3dbc0601f8c2a0e696441" Oct 31 21:37:11.502185 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:11.502116 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d63cbad-61eb-4678-abe6-02ee5810d710" path="/var/lib/kubelet/pods/6d63cbad-61eb-4678-abe6-02ee5810d710/volumes" Oct 31 21:37:13.032063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:13.032027 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq"] Oct 31 21:37:13.041399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:13.041359 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54713ff3-ftbmq"] Oct 31 21:37:13.475666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:13.475629 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0205e11-1145-42d4-952d-66e7b58e8416" path="/var/lib/kubelet/pods/d0205e11-1145-42d4-952d-66e7b58e8416/volumes" Oct 31 21:37:19.034072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:19.034034 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2"] Oct 31 21:37:19.044268 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:19.044224 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d3859a-pd7h2"] Oct 31 21:37:19.476671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:19.476633 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a8d24e-418d-4e8d-86db-2e3f683dc435" path="/var/lib/kubelet/pods/11a8d24e-418d-4e8d-86db-2e3f683dc435/volumes" Oct 31 21:37:32.604943 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:37:32.604900 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:37:33.033435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:33.033396 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9"] Oct 31 21:37:33.043138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:33.043104 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f8127c6-px9s9"] Oct 31 21:37:33.475989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:33.475956 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0289b214-3ddd-4788-9efe-ae79885c6003" path="/var/lib/kubelet/pods/0289b214-3ddd-4788-9efe-ae79885c6003/volumes" Oct 31 21:37:34.034719 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:34.034631 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz"] Oct 31 21:37:34.045657 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:34.045618 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ce8caab-s2rkz"] Oct 31 21:37:35.477216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:35.477170 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79b16cd-9abb-42bc-950e-5c7f4bd0397f" path="/var/lib/kubelet/pods/f79b16cd-9abb-42bc-950e-5c7f4bd0397f/volumes" Oct 31 21:37:39.039508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:39.039470 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg"] Oct 31 21:37:39.050219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:39.050182 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8426f638-2psbg"] Oct 31 21:37:39.476307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:39.476270 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6082f947-acba-4b6e-8945-def617bcbc54" path="/var/lib/kubelet/pods/6082f947-acba-4b6e-8945-def617bcbc54/volumes" Oct 31 21:37:40.221708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:40.221661 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x"] Oct 31 21:37:40.228595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:40.228565 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-43fb1b-j9p5x"] Oct 31 21:37:41.474544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:41.474502 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9649d783-a478-4ee8-8de0-97573500d2de" path="/var/lib/kubelet/pods/9649d783-a478-4ee8-8de0-97573500d2de/volumes" Oct 31 21:37:54.726594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.726556 5188 scope.go:122] "RemoveContainer" containerID="a5709e82c3419c453403f480f7be64a214bfc7744b29d2504e0d0240de2d2e53" Oct 31 21:37:54.738475 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.738450 5188 scope.go:122] "RemoveContainer" containerID="b1ce2548e8b090259e466a2de591921c12aad51abee2732f43af9be9ed2ccd7a" Oct 31 21:37:54.749708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.749680 5188 scope.go:122] "RemoveContainer" containerID="c09bd1371af7437c2cd0797c7a6e6940afae73fc618f3e6c0ada884beac6b50e" Oct 31 21:37:54.761242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.761219 5188 scope.go:122] "RemoveContainer" containerID="3f8c506609f1174e513c1d86267049c5b655c89617c23fbe4c94a625be91eff1" Oct 31 21:37:54.775442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.775411 5188 scope.go:122] "RemoveContainer" containerID="de6831a34757c92adba71bbe42b6b6975aef628cd352f7dace9a0299204da069" Oct 31 21:37:54.787016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.786997 5188 scope.go:122] "RemoveContainer" containerID="7e18d40d5c93c802e1b7954b5839179d1204800199196935453a0358da4669c7" Oct 31 21:37:54.799763 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.799737 5188 scope.go:122] "RemoveContainer" containerID="b1fdb5df24b735ddfb6a68f100665caeea6d0afb8aa4a1f62b78f55a18063afc" Oct 31 21:37:54.810950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.810929 5188 scope.go:122] "RemoveContainer" containerID="c8b27fe69b0db8b8f0058d5ce95028e6f16462495a150a1337429d46c271c535" Oct 31 21:37:54.825317 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.825178 5188 scope.go:122] "RemoveContainer" containerID="5f7886601fe680b4b8a4c243020f457fb115018ac93d7d811bb37794242c1f64" Oct 31 21:37:54.839506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:54.839479 5188 scope.go:122] "RemoveContainer" containerID="d2ecc52f9c0bc3a56a860297ef9e3686c62ab8fb9e52adf7fcc670f547efa614" Oct 31 21:37:59.036561 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:59.036526 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26"] Oct 31 21:37:59.045364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:59.045312 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598d3869-h5b26"] Oct 31 21:37:59.474627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:37:59.474580 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35627e3c-c0a8-48a0-9bfd-9732d2385ebd" path="/var/lib/kubelet/pods/35627e3c-c0a8-48a0-9bfd-9732d2385ebd/volumes" Oct 31 21:38:00.921767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:00.921727 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh"] Oct 31 21:38:00.922692 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:00.922636 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" containerID="cri-o://58d84e002395f43f5559ec4c46453561b40d9f8f8d93d96221f10114daadcd05" gracePeriod=30 Oct 31 21:38:02.589086 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:38:02.589036 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:38:03.039037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:03.039001 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk"] Oct 31 21:38:03.047085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:03.047053 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc147650-l69pk"] Oct 31 21:38:03.545346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:03.541907 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62385c77-0dc0-43b8-b4d0-a6fd89b2bb45" path="/var/lib/kubelet/pods/62385c77-0dc0-43b8-b4d0-a6fd89b2bb45/volumes" Oct 31 21:38:06.185266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.185019 5188 generic.go:356] "Generic (PLEG): container finished" podID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerID="58d84e002395f43f5559ec4c46453561b40d9f8f8d93d96221f10114daadcd05" exitCode=0 Oct 31 21:38:06.185266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.185084 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" event={"ID":"96111e4f-31bc-4769-a00a-ddf36fa4e2af","Type":"ContainerDied","Data":"58d84e002395f43f5559ec4c46453561b40d9f8f8d93d96221f10114daadcd05"} Oct 31 21:38:06.185266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.185145 5188 scope.go:122] "RemoveContainer" containerID="a6f0d37a791933965eb010150c0988434006bc639ee6ea88a596c3cd10bced1c" Oct 31 21:38:06.406807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.406765 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:38:06.517099 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.516984 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-kube-api-access-hxvm4\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-kube-api-access-hxvm4\") pod \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " Oct 31 21:38:06.517099 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517044 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-credential-socket\") pod \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " Oct 31 21:38:06.517435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517172 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-data\") pod \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " Oct 31 21:38:06.517435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517286 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istiod-ca-cert\") pod \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " Oct 31 21:38:06.517435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517344 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-token\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-token\") pod \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " Oct 31 21:38:06.517435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517370 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-credential-socket" pod "96111e4f-31bc-4769-a00a-ddf36fa4e2af" (UID: "96111e4f-31bc-4769-a00a-ddf36fa4e2af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.517435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517391 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-socket\") pod \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " Oct 31 21:38:06.517715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517449 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-podinfo\") pod \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " Oct 31 21:38:06.517715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517548 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-envoy\") pod \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " Oct 31 21:38:06.517715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517654 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-certs\") pod \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\" (UID: \"96111e4f-31bc-4769-a00a-ddf36fa4e2af\") " Oct 31 21:38:06.517715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517701 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istiod-ca-cert" pod "96111e4f-31bc-4769-a00a-ddf36fa4e2af" (UID: "96111e4f-31bc-4769-a00a-ddf36fa4e2af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:06.517983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.517912 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-data" pod "96111e4f-31bc-4769-a00a-ddf36fa4e2af" (UID: "96111e4f-31bc-4769-a00a-ddf36fa4e2af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.518088 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.518068 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-socket" pod "96111e4f-31bc-4769-a00a-ddf36fa4e2af" (UID: "96111e4f-31bc-4769-a00a-ddf36fa4e2af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.518488 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.518464 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.518667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.518651 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.518766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.518744 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-certs" pod "96111e4f-31bc-4769-a00a-ddf36fa4e2af" (UID: "96111e4f-31bc-4769-a00a-ddf36fa4e2af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.518855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.518762 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.518952 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.518938 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.521166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.521139 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-envoy" pod "96111e4f-31bc-4769-a00a-ddf36fa4e2af" (UID: "96111e4f-31bc-4769-a00a-ddf36fa4e2af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.521911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.521883 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-kube-api-access-hxvm4" pod "96111e4f-31bc-4769-a00a-ddf36fa4e2af" (UID: "96111e4f-31bc-4769-a00a-ddf36fa4e2af"). InnerVolumeSpecName "kube-api-access-hxvm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:06.522923 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.522897 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-podinfo" pod "96111e4f-31bc-4769-a00a-ddf36fa4e2af" (UID: "96111e4f-31bc-4769-a00a-ddf36fa4e2af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:06.523300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.523276 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-token" pod "96111e4f-31bc-4769-a00a-ddf36fa4e2af" (UID: "96111e4f-31bc-4769-a00a-ddf36fa4e2af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:06.619974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.619928 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.619974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.619970 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxvm4\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-kube-api-access-hxvm4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.619974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.619981 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.620230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.619990 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.620230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:06.619998 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96111e4f-31bc-4769-a00a-ddf36fa4e2af-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:07.198661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:07.198627 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" Oct 31 21:38:07.199133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:07.198663 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh" event={"ID":"96111e4f-31bc-4769-a00a-ddf36fa4e2af","Type":"ContainerDied","Data":"fa3249b239d977625eece916fc0e5037f9b003c20dd2b51daadb079eef72da9c"} Oct 31 21:38:07.199133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:07.198736 5188 scope.go:122] "RemoveContainer" containerID="58d84e002395f43f5559ec4c46453561b40d9f8f8d93d96221f10114daadcd05" Oct 31 21:38:07.253070 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:07.253027 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh"] Oct 31 21:38:07.258758 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:07.258720 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-zqhjh"] Oct 31 21:38:07.480092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:07.479088 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" path="/var/lib/kubelet/pods/96111e4f-31bc-4769-a00a-ddf36fa4e2af/volumes" Oct 31 21:38:15.145047 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:15.145001 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p"] Oct 31 21:38:15.145581 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:15.145307 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" containerID="cri-o://75349768e0454cb9378a9e43b76199ed45cf679a5c55460f81517d3c93f9d3fd" gracePeriod=30 Oct 31 21:38:15.155020 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:15.154750 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2"] Oct 31 21:38:15.155230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:15.155089 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" containerID="cri-o://bbecb38fc37676bc8a76ec2d348d488b31701bdf5148de4994600bc9f376ac65" gracePeriod=30 Oct 31 21:38:15.160675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:15.160631 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76"] Oct 31 21:38:15.304042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:15.303989 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" containerID="cri-o://c37842e167782b64f36a50ea205bd1daa30893f2394de99e8255e8c38dc18b0b" gracePeriod=30 Oct 31 21:38:16.060011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:16.059965 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx"] Oct 31 21:38:16.066676 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:16.066644 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5874c9d2-tgrgx"] Oct 31 21:38:17.474882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:17.474843 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba70b88d-77ab-49f2-b66b-2b005d8962ac" path="/var/lib/kubelet/pods/ba70b88d-77ab-49f2-b66b-2b005d8962ac/volumes" Oct 31 21:38:19.514542 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:19.514450 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk"] Oct 31 21:38:19.515157 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:19.514873 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" containerID="cri-o://a82dc61a344c5350fd08c6677630453f34b0fc3b086533ae1ddd257c75ebd116" gracePeriod=30 Oct 31 21:38:19.524251 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:19.524201 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c"] Oct 31 21:38:19.524562 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:19.524532 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" containerID="cri-o://9bd82551b7551f90adbb6e2a89f4bf5c7d87e4a8896ff02b536a05a333451832" gracePeriod=30 Oct 31 21:38:19.528790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:19.528757 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96"] Oct 31 21:38:19.528993 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:19.528973 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" containerID="cri-o://0e36e3cc2c63c78607c6637f75b461a368c97cfc1d52e518b199d19647a6cfc6" gracePeriod=30 Oct 31 21:38:19.537989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:19.537951 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d"] Oct 31 21:38:19.538675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:19.538609 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" containerID="cri-o://962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd" gracePeriod=30 Oct 31 21:38:20.139532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.139485 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7"] Oct 31 21:38:20.140203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.140185 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9649d783-a478-4ee8-8de0-97573500d2de" containerName="revoke-keys" Oct 31 21:38:20.140257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.140205 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9649d783-a478-4ee8-8de0-97573500d2de" containerName="revoke-keys" Oct 31 21:38:20.140257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.140218 5188 cpu_manager.go:414] "containerMap: removing container" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" Oct 31 21:38:20.140257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.140223 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" Oct 31 21:38:20.140257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.140257 5188 cpu_manager.go:414] "containerMap: removing container" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" Oct 31 21:38:20.140397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.140263 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" Oct 31 21:38:20.140397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.140374 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" Oct 31 21:38:20.140397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.140386 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9649d783-a478-4ee8-8de0-97573500d2de" containerName="revoke-keys" Oct 31 21:38:20.145083 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.145054 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:20.162462 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.162381 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d75c94\"" Oct 31 21:38:20.167291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.162946 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d75c94-dockercfg-bx6m6\"" Oct 31 21:38:20.167291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.165958 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7"] Oct 31 21:38:20.224566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.224529 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2"] Oct 31 21:38:20.226101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.226065 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="96111e4f-31bc-4769-a00a-ddf36fa4e2af" containerName="istio-proxy" Oct 31 21:38:20.231137 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.231098 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee63ab77-96a8-4c13-828a-49554f3ef33a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7\" (UID: \"ee63ab77-96a8-4c13-828a-49554f3ef33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:20.231424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.231242 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prsjh\" (UniqueName: \"kubernetes.io/projected/ee63ab77-96a8-4c13-828a-49554f3ef33a-kube-api-access-prsjh\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7\" (UID: \"ee63ab77-96a8-4c13-828a-49554f3ef33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:20.234649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.234620 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:20.264346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.252134 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2"] Oct 31 21:38:20.357553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.332452 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2\" (UID: \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:20.422623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.403707 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee63ab77-96a8-4c13-828a-49554f3ef33a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7\" (UID: \"ee63ab77-96a8-4c13-828a-49554f3ef33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:20.422623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.403828 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxqt\" (UniqueName: \"kubernetes.io/projected/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-kube-api-access-snxqt\") pod \"payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2\" (UID: \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:20.422623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.403910 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prsjh\" (UniqueName: \"kubernetes.io/projected/ee63ab77-96a8-4c13-828a-49554f3ef33a-kube-api-access-prsjh\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7\" (UID: \"ee63ab77-96a8-4c13-828a-49554f3ef33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:20.422623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.405576 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee63ab77-96a8-4c13-828a-49554f3ef33a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7\" (UID: \"ee63ab77-96a8-4c13-828a-49554f3ef33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:20.441657 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.441621 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prsjh\" (UniqueName: \"kubernetes.io/projected/ee63ab77-96a8-4c13-828a-49554f3ef33a-kube-api-access-prsjh\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7\" (UID: \"ee63ab77-96a8-4c13-828a-49554f3ef33a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:20.470759 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.470713 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:20.530115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.508361 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxqt\" (UniqueName: \"kubernetes.io/projected/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-kube-api-access-snxqt\") pod \"payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2\" (UID: \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:20.530115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.508635 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2\" (UID: \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:20.530115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.509712 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2\" (UID: \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:20.544412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.544041 5188 generic.go:356] "Generic (PLEG): container finished" podID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerID="c37842e167782b64f36a50ea205bd1daa30893f2394de99e8255e8c38dc18b0b" exitCode=0 Oct 31 21:38:20.544412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.544288 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" event={"ID":"c36232b3-9979-41d1-97fe-518e3a25ac76","Type":"ContainerDied","Data":"c37842e167782b64f36a50ea205bd1daa30893f2394de99e8255e8c38dc18b0b"} Oct 31 21:38:20.546954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.546884 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxqt\" (UniqueName: \"kubernetes.io/projected/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-kube-api-access-snxqt\") pod \"payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2\" (UID: \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:20.564804 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.564439 5188 generic.go:356] "Generic (PLEG): container finished" podID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerID="75349768e0454cb9378a9e43b76199ed45cf679a5c55460f81517d3c93f9d3fd" exitCode=0 Oct 31 21:38:20.564804 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.564712 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" event={"ID":"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b","Type":"ContainerDied","Data":"75349768e0454cb9378a9e43b76199ed45cf679a5c55460f81517d3c93f9d3fd"} Oct 31 21:38:20.589374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.589263 5188 generic.go:356] "Generic (PLEG): container finished" podID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerID="bbecb38fc37676bc8a76ec2d348d488b31701bdf5148de4994600bc9f376ac65" exitCode=0 Oct 31 21:38:20.589770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.589606 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" event={"ID":"776d7137-587e-4c9a-8fbb-648a39b77e34","Type":"ContainerDied","Data":"bbecb38fc37676bc8a76ec2d348d488b31701bdf5148de4994600bc9f376ac65"} Oct 31 21:38:20.834944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.834823 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:20.843356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.841160 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:38:20.932352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.930948 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-envoy\") pod \"c36232b3-9979-41d1-97fe-518e3a25ac76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " Oct 31 21:38:20.932352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.931034 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-podinfo\") pod \"c36232b3-9979-41d1-97fe-518e3a25ac76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " Oct 31 21:38:20.932352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.931064 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-credential-socket\") pod \"c36232b3-9979-41d1-97fe-518e3a25ac76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " Oct 31 21:38:20.932352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.931171 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-certs\") pod \"c36232b3-9979-41d1-97fe-518e3a25ac76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " Oct 31 21:38:20.932352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.931200 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-token\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-token\") pod \"c36232b3-9979-41d1-97fe-518e3a25ac76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " Oct 31 21:38:20.932352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.931219 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-kube-api-access-6kksc\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-kube-api-access-6kksc\") pod \"c36232b3-9979-41d1-97fe-518e3a25ac76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " Oct 31 21:38:20.932352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.931242 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-socket\") pod \"c36232b3-9979-41d1-97fe-518e3a25ac76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " Oct 31 21:38:20.932352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.931301 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c36232b3-9979-41d1-97fe-518e3a25ac76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36232b3-9979-41d1-97fe-518e3a25ac76-istiod-ca-cert\") pod \"c36232b3-9979-41d1-97fe-518e3a25ac76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " Oct 31 21:38:20.932352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.931414 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-data\") pod \"c36232b3-9979-41d1-97fe-518e3a25ac76\" (UID: \"c36232b3-9979-41d1-97fe-518e3a25ac76\") " Oct 31 21:38:20.936302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.933015 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-data" pod "c36232b3-9979-41d1-97fe-518e3a25ac76" (UID: "c36232b3-9979-41d1-97fe-518e3a25ac76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:20.936302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.936074 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-token" pod "c36232b3-9979-41d1-97fe-518e3a25ac76" (UID: "c36232b3-9979-41d1-97fe-518e3a25ac76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:20.938337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.936832 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-socket" pod "c36232b3-9979-41d1-97fe-518e3a25ac76" (UID: "c36232b3-9979-41d1-97fe-518e3a25ac76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:20.938337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.936919 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c36232b3-9979-41d1-97fe-518e3a25ac76-istiod-ca-cert" pod "c36232b3-9979-41d1-97fe-518e3a25ac76" (UID: "c36232b3-9979-41d1-97fe-518e3a25ac76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:20.938337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.937465 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-credential-socket" pod "c36232b3-9979-41d1-97fe-518e3a25ac76" (UID: "c36232b3-9979-41d1-97fe-518e3a25ac76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:20.938337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.937542 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-certs" pod "c36232b3-9979-41d1-97fe-518e3a25ac76" (UID: "c36232b3-9979-41d1-97fe-518e3a25ac76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:20.939663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.939124 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-kube-api-access-6kksc" pod "c36232b3-9979-41d1-97fe-518e3a25ac76" (UID: "c36232b3-9979-41d1-97fe-518e3a25ac76"). InnerVolumeSpecName "kube-api-access-6kksc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:20.941379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.940525 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-envoy" pod "c36232b3-9979-41d1-97fe-518e3a25ac76" (UID: "c36232b3-9979-41d1-97fe-518e3a25ac76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:20.943794 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.943762 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-podinfo" pod "c36232b3-9979-41d1-97fe-518e3a25ac76" (UID: "c36232b3-9979-41d1-97fe-518e3a25ac76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:20.946229 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.945874 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:38:20.965469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:20.964841 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:38:21.033510 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.033463 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-data\") pod \"776d7137-587e-4c9a-8fbb-648a39b77e34\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " Oct 31 21:38:21.033732 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.033546 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-envoy\") pod \"776d7137-587e-4c9a-8fbb-648a39b77e34\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " Oct 31 21:38:21.033732 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.033682 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-podinfo\") pod \"776d7137-587e-4c9a-8fbb-648a39b77e34\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " Oct 31 21:38:21.033732 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.033717 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-credential-socket\") pod \"776d7137-587e-4c9a-8fbb-648a39b77e34\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " Oct 31 21:38:21.033912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.033780 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-kube-api-access-sqskw\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-kube-api-access-sqskw\") pod \"776d7137-587e-4c9a-8fbb-648a39b77e34\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " Oct 31 21:38:21.033912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.033814 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776d7137-587e-4c9a-8fbb-648a39b77e34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776d7137-587e-4c9a-8fbb-648a39b77e34-istiod-ca-cert\") pod \"776d7137-587e-4c9a-8fbb-648a39b77e34\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " Oct 31 21:38:21.033912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.033852 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-token\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-token\") pod \"776d7137-587e-4c9a-8fbb-648a39b77e34\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " Oct 31 21:38:21.034179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.033914 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-socket\") pod \"776d7137-587e-4c9a-8fbb-648a39b77e34\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " Oct 31 21:38:21.034179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.034068 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-certs\") pod \"776d7137-587e-4c9a-8fbb-648a39b77e34\" (UID: \"776d7137-587e-4c9a-8fbb-648a39b77e34\") " Oct 31 21:38:21.036172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.035427 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.036172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.035474 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.036172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.035491 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kksc\" (UniqueName: \"kubernetes.io/projected/c36232b3-9979-41d1-97fe-518e3a25ac76-kube-api-access-6kksc\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.036172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.035506 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.036172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.035524 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c36232b3-9979-41d1-97fe-518e3a25ac76-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.036172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.035539 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.036172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.035565 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.036172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.035579 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c36232b3-9979-41d1-97fe-518e3a25ac76-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.036172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.035592 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c36232b3-9979-41d1-97fe-518e3a25ac76-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.037494 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.037458 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776d7137-587e-4c9a-8fbb-648a39b77e34-istiod-ca-cert" pod "776d7137-587e-4c9a-8fbb-648a39b77e34" (UID: "776d7137-587e-4c9a-8fbb-648a39b77e34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:21.039289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.039208 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-certs" pod "776d7137-587e-4c9a-8fbb-648a39b77e34" (UID: "776d7137-587e-4c9a-8fbb-648a39b77e34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.039289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.039242 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-credential-socket" pod "776d7137-587e-4c9a-8fbb-648a39b77e34" (UID: "776d7137-587e-4c9a-8fbb-648a39b77e34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.039488 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.039311 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-data" pod "776d7137-587e-4c9a-8fbb-648a39b77e34" (UID: "776d7137-587e-4c9a-8fbb-648a39b77e34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.039642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.039606 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-socket" pod "776d7137-587e-4c9a-8fbb-648a39b77e34" (UID: "776d7137-587e-4c9a-8fbb-648a39b77e34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.042847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.042798 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-token" pod "776d7137-587e-4c9a-8fbb-648a39b77e34" (UID: "776d7137-587e-4c9a-8fbb-648a39b77e34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:21.043101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.043058 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-podinfo" pod "776d7137-587e-4c9a-8fbb-648a39b77e34" (UID: "776d7137-587e-4c9a-8fbb-648a39b77e34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:21.044857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.044814 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-kube-api-access-sqskw" pod "776d7137-587e-4c9a-8fbb-648a39b77e34" (UID: "776d7137-587e-4c9a-8fbb-648a39b77e34"). InnerVolumeSpecName "kube-api-access-sqskw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:21.045971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.045932 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-envoy" pod "776d7137-587e-4c9a-8fbb-648a39b77e34" (UID: "776d7137-587e-4c9a-8fbb-648a39b77e34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.054437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.054389 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww"] Oct 31 21:38:21.061447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.061404 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1870b67-bkzww"] Oct 31 21:38:21.125111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.125079 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7"] Oct 31 21:38:21.125395 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:21.125369 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee63ab77_96a8_4c13_828a_49554f3ef33a.slice/crio-23e94c03d0be6de767c4c3550b241b71b88e9270195f488fda1fd2ee1b0e0dad WatchSource:0}: Error finding container 23e94c03d0be6de767c4c3550b241b71b88e9270195f488fda1fd2ee1b0e0dad: Status 404 returned error can't find the container with id 23e94c03d0be6de767c4c3550b241b71b88e9270195f488fda1fd2ee1b0e0dad Oct 31 21:38:21.137082 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137041 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-credential-socket\") pod \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " Oct 31 21:38:21.137261 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137167 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-certs\") pod \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " Oct 31 21:38:21.137261 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137202 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-data\") pod \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " Oct 31 21:38:21.137574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137300 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-podinfo\") pod \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " Oct 31 21:38:21.137574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137374 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-credential-socket" pod "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" (UID: "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.137574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137392 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istiod-ca-cert\") pod \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " Oct 31 21:38:21.137574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137472 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-token\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-token\") pod \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " Oct 31 21:38:21.137574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137548 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-data" pod "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" (UID: "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.137574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137568 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-envoy\") pod \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " Oct 31 21:38:21.137902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137822 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-socket\") pod \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " Oct 31 21:38:21.137902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137563 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-certs" pod "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" (UID: "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.137902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137776 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istiod-ca-cert" pod "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" (UID: "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:21.137902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.137874 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-kube-api-access-fffft\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-kube-api-access-fffft\") pod \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\" (UID: \"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b\") " Oct 31 21:38:21.138436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138407 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-socket" pod "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" (UID: "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.138663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138641 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.138753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138666 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.138753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138685 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.138753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138700 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.138753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138716 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.138753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138728 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.138753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138742 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.139084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138756 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.139084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138770 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.139084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138786 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.139084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138800 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqskw\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-kube-api-access-sqskw\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.139084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138814 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776d7137-587e-4c9a-8fbb-648a39b77e34-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.139084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138828 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776d7137-587e-4c9a-8fbb-648a39b77e34-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.139084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.138841 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776d7137-587e-4c9a-8fbb-648a39b77e34-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.140173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.140144 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-kube-api-access-fffft" pod "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" (UID: "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b"). InnerVolumeSpecName "kube-api-access-fffft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:21.140352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.140296 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-envoy" pod "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" (UID: "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:21.140431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.140298 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-podinfo" pod "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" (UID: "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:21.140467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.140455 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-token" pod "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" (UID: "a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:21.207679 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:21.207641 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f9027e_c1b9_46b6_bc5a_fb827fc67c6f.slice/crio-dbebbc27c7caf9f69d16530172bedb9c71926812550e6a471ad482a4b5e04c09 WatchSource:0}: Error finding container dbebbc27c7caf9f69d16530172bedb9c71926812550e6a471ad482a4b5e04c09: Status 404 returned error can't find the container with id dbebbc27c7caf9f69d16530172bedb9c71926812550e6a471ad482a4b5e04c09 Oct 31 21:38:21.208504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.208476 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2"] Oct 31 21:38:21.240166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.240131 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.240166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.240161 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.240166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.240170 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.240475 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.240179 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fffft\" (UniqueName: \"kubernetes.io/projected/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b-kube-api-access-fffft\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:21.475081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.474972 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9883fcca-9fdf-438f-9688-a435b5154ce4" path="/var/lib/kubelet/pods/9883fcca-9fdf-438f-9688-a435b5154ce4/volumes" Oct 31 21:38:21.601927 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.601890 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" Oct 31 21:38:21.602465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.601898 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2" event={"ID":"776d7137-587e-4c9a-8fbb-648a39b77e34","Type":"ContainerDied","Data":"214fde7249b6640f4512fc928a2d2a8803bfb7e3ae6fcc6a7b173e860cb275fa"} Oct 31 21:38:21.602465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.602056 5188 scope.go:122] "RemoveContainer" containerID="bbecb38fc37676bc8a76ec2d348d488b31701bdf5148de4994600bc9f376ac65" Oct 31 21:38:21.606937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.606877 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" event={"ID":"c36232b3-9979-41d1-97fe-518e3a25ac76","Type":"ContainerDied","Data":"f8e04d7522c8495d3f908743b53a9501b7b283a123c6243d03848273916d40ff"} Oct 31 21:38:21.607097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.607057 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76" Oct 31 21:38:21.610295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.610229 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" event={"ID":"ee63ab77-96a8-4c13-828a-49554f3ef33a","Type":"ContainerStarted","Data":"17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47"} Oct 31 21:38:21.610529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.610387 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" event={"ID":"ee63ab77-96a8-4c13-828a-49554f3ef33a","Type":"ContainerStarted","Data":"23e94c03d0be6de767c4c3550b241b71b88e9270195f488fda1fd2ee1b0e0dad"} Oct 31 21:38:21.610649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.610633 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:21.613136 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.613109 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" Oct 31 21:38:21.613299 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.613129 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p" event={"ID":"a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b","Type":"ContainerDied","Data":"cd057a13eb8c593ce385d6a9fe05302681d30a40b6331d42ca120ae48a991355"} Oct 31 21:38:21.616070 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.616035 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" event={"ID":"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f","Type":"ContainerStarted","Data":"6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be"} Oct 31 21:38:21.616284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.616093 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" event={"ID":"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f","Type":"ContainerStarted","Data":"dbebbc27c7caf9f69d16530172bedb9c71926812550e6a471ad482a4b5e04c09"} Oct 31 21:38:21.617091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.617060 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:21.623206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.623173 5188 scope.go:122] "RemoveContainer" containerID="c37842e167782b64f36a50ea205bd1daa30893f2394de99e8255e8c38dc18b0b" Oct 31 21:38:21.636158 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.636129 5188 scope.go:122] "RemoveContainer" containerID="75349768e0454cb9378a9e43b76199ed45cf679a5c55460f81517d3c93f9d3fd" Oct 31 21:38:21.655189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.655123 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2"] Oct 31 21:38:21.660379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.660311 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-rvcb2"] Oct 31 21:38:21.704886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.704847 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76"] Oct 31 21:38:21.718479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.718431 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-4zj76"] Oct 31 21:38:21.745160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.745045 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" podStartSLOduration=1.7450273360000002 podStartE2EDuration="1.745027336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:21.741789508 +0000 UTC m=+1348.434746901" watchObservedRunningTime="2026-10-31 21:38:21.745027336 +0000 UTC m=+1348.437984720" Oct 31 21:38:21.778204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.778161 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p"] Oct 31 21:38:21.786922 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.786874 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-mht2p"] Oct 31 21:38:21.820813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:21.820749 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" podStartSLOduration=1.820733459 podStartE2EDuration="1.820733459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:21.818694809 +0000 UTC m=+1348.511652203" watchObservedRunningTime="2026-10-31 21:38:21.820733459 +0000 UTC m=+1348.513690851" Oct 31 21:38:23.477501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:23.477460 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" path="/var/lib/kubelet/pods/776d7137-587e-4c9a-8fbb-648a39b77e34/volumes" Oct 31 21:38:23.478198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:23.478174 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" path="/var/lib/kubelet/pods/a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b/volumes" Oct 31 21:38:23.478758 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:23.478736 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" path="/var/lib/kubelet/pods/c36232b3-9979-41d1-97fe-518e3a25ac76/volumes" Oct 31 21:38:24.653253 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.653215 5188 generic.go:356] "Generic (PLEG): container finished" podID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerID="9bd82551b7551f90adbb6e2a89f4bf5c7d87e4a8896ff02b536a05a333451832" exitCode=0 Oct 31 21:38:24.653854 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.653288 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" event={"ID":"94bf0314-53b2-4937-9106-f72080e6d0f1","Type":"ContainerDied","Data":"9bd82551b7551f90adbb6e2a89f4bf5c7d87e4a8896ff02b536a05a333451832"} Oct 31 21:38:24.656610 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.656473 5188 generic.go:356] "Generic (PLEG): container finished" podID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerID="a82dc61a344c5350fd08c6677630453f34b0fc3b086533ae1ddd257c75ebd116" exitCode=0 Oct 31 21:38:24.656610 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.656542 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" event={"ID":"08e1f58a-9117-4538-abe5-c8377b9f5434","Type":"ContainerDied","Data":"a82dc61a344c5350fd08c6677630453f34b0fc3b086533ae1ddd257c75ebd116"} Oct 31 21:38:24.659557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.659475 5188 generic.go:356] "Generic (PLEG): container finished" podID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerID="0e36e3cc2c63c78607c6637f75b461a368c97cfc1d52e518b199d19647a6cfc6" exitCode=0 Oct 31 21:38:24.659682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.659558 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" event={"ID":"2829db12-be6a-46ba-a4d9-58158af0f3b4","Type":"ContainerDied","Data":"0e36e3cc2c63c78607c6637f75b461a368c97cfc1d52e518b199d19647a6cfc6"} Oct 31 21:38:24.659682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.659610 5188 scope.go:122] "RemoveContainer" containerID="2205f7189c45a4ae05943c1a68ec455137ef1cbfcc0f9eaa38e43cbb9e58760c" Oct 31 21:38:24.956539 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.956510 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:38:24.971281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.971245 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:38:24.995971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:24.995941 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:38:25.031163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.031127 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-socket\") pod \"4e0af82a-8a48-4758-97ca-48ddb633fda8\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " Oct 31 21:38:25.032050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.031399 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-socket" pod "4e0af82a-8a48-4758-97ca-48ddb633fda8" (UID: "4e0af82a-8a48-4758-97ca-48ddb633fda8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.032050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.031475 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-data\") pod \"2829db12-be6a-46ba-a4d9-58158af0f3b4\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " Oct 31 21:38:25.032050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.031600 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-envoy\") pod \"94bf0314-53b2-4937-9106-f72080e6d0f1\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " Oct 31 21:38:25.032050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.031647 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-credential-socket\") pod \"2829db12-be6a-46ba-a4d9-58158af0f3b4\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " Oct 31 21:38:25.032050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.031692 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-data" pod "2829db12-be6a-46ba-a4d9-58158af0f3b4" (UID: "2829db12-be6a-46ba-a4d9-58158af0f3b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.032050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.031695 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-token\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-token\") pod \"4e0af82a-8a48-4758-97ca-48ddb633fda8\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032278 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-credential-socket" pod "2829db12-be6a-46ba-a4d9-58158af0f3b4" (UID: "2829db12-be6a-46ba-a4d9-58158af0f3b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032354 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-data\") pod \"4e0af82a-8a48-4758-97ca-48ddb633fda8\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032425 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-token\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-token\") pod \"2829db12-be6a-46ba-a4d9-58158af0f3b4\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032467 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-certs\") pod \"4e0af82a-8a48-4758-97ca-48ddb633fda8\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032507 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-data\") pod \"94bf0314-53b2-4937-9106-f72080e6d0f1\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032540 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-podinfo\") pod \"4e0af82a-8a48-4758-97ca-48ddb633fda8\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032587 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-podinfo\") pod \"94bf0314-53b2-4937-9106-f72080e6d0f1\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032613 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e0af82a-8a48-4758-97ca-48ddb633fda8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0af82a-8a48-4758-97ca-48ddb633fda8-istiod-ca-cert\") pod \"4e0af82a-8a48-4758-97ca-48ddb633fda8\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032642 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-kube-api-access-zhh6k\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-kube-api-access-zhh6k\") pod \"4e0af82a-8a48-4758-97ca-48ddb633fda8\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032680 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-socket\") pod \"2829db12-be6a-46ba-a4d9-58158af0f3b4\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032716 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94bf0314-53b2-4937-9106-f72080e6d0f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bf0314-53b2-4937-9106-f72080e6d0f1-istiod-ca-cert\") pod \"94bf0314-53b2-4937-9106-f72080e6d0f1\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032750 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-credential-socket\") pod \"4e0af82a-8a48-4758-97ca-48ddb633fda8\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032804 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-certs\") pod \"94bf0314-53b2-4937-9106-f72080e6d0f1\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " Oct 31 21:38:25.032842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032844 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2829db12-be6a-46ba-a4d9-58158af0f3b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2829db12-be6a-46ba-a4d9-58158af0f3b4-istiod-ca-cert\") pod \"2829db12-be6a-46ba-a4d9-58158af0f3b4\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " Oct 31 21:38:25.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032892 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-socket\") pod \"94bf0314-53b2-4937-9106-f72080e6d0f1\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " Oct 31 21:38:25.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032946 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-certs\") pod \"2829db12-be6a-46ba-a4d9-58158af0f3b4\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " Oct 31 21:38:25.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.032994 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-kube-api-access-hdvfg\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-kube-api-access-hdvfg\") pod \"2829db12-be6a-46ba-a4d9-58158af0f3b4\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " Oct 31 21:38:25.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.033065 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-kube-api-access-kj58w\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-kube-api-access-kj58w\") pod \"94bf0314-53b2-4937-9106-f72080e6d0f1\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " Oct 31 21:38:25.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.033121 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-credential-socket\") pod \"94bf0314-53b2-4937-9106-f72080e6d0f1\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " Oct 31 21:38:25.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.033155 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-podinfo\") pod \"2829db12-be6a-46ba-a4d9-58158af0f3b4\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " Oct 31 21:38:25.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.033213 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-envoy\") pod \"2829db12-be6a-46ba-a4d9-58158af0f3b4\" (UID: \"2829db12-be6a-46ba-a4d9-58158af0f3b4\") " Oct 31 21:38:25.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.033244 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-token\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-token\") pod \"94bf0314-53b2-4937-9106-f72080e6d0f1\" (UID: \"94bf0314-53b2-4937-9106-f72080e6d0f1\") " Oct 31 21:38:25.033549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.033293 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-envoy\") pod \"4e0af82a-8a48-4758-97ca-48ddb633fda8\" (UID: \"4e0af82a-8a48-4758-97ca-48ddb633fda8\") " Oct 31 21:38:25.035308 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.035278 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-credential-socket" pod "4e0af82a-8a48-4758-97ca-48ddb633fda8" (UID: "4e0af82a-8a48-4758-97ca-48ddb633fda8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.035621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.035595 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-socket" pod "2829db12-be6a-46ba-a4d9-58158af0f3b4" (UID: "2829db12-be6a-46ba-a4d9-58158af0f3b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.036469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.036432 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.036469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.036462 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.036623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.036477 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.036623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.036492 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.036623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.036505 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.036817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.036776 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-envoy" pod "94bf0314-53b2-4937-9106-f72080e6d0f1" (UID: "94bf0314-53b2-4937-9106-f72080e6d0f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.037703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.037054 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-certs" pod "4e0af82a-8a48-4758-97ca-48ddb633fda8" (UID: "4e0af82a-8a48-4758-97ca-48ddb633fda8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.037703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.037332 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0af82a-8a48-4758-97ca-48ddb633fda8-istiod-ca-cert" pod "4e0af82a-8a48-4758-97ca-48ddb633fda8" (UID: "4e0af82a-8a48-4758-97ca-48ddb633fda8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:25.037703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.037428 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-podinfo" pod "4e0af82a-8a48-4758-97ca-48ddb633fda8" (UID: "4e0af82a-8a48-4758-97ca-48ddb633fda8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:25.037940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.037737 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-data" pod "4e0af82a-8a48-4758-97ca-48ddb633fda8" (UID: "4e0af82a-8a48-4758-97ca-48ddb633fda8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.038004 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.037959 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94bf0314-53b2-4937-9106-f72080e6d0f1-istiod-ca-cert" pod "94bf0314-53b2-4937-9106-f72080e6d0f1" (UID: "94bf0314-53b2-4937-9106-f72080e6d0f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:25.039534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.039201 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-data" pod "94bf0314-53b2-4937-9106-f72080e6d0f1" (UID: "94bf0314-53b2-4937-9106-f72080e6d0f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.039534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.039459 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-kube-api-access-zhh6k" pod "4e0af82a-8a48-4758-97ca-48ddb633fda8" (UID: "4e0af82a-8a48-4758-97ca-48ddb633fda8"). InnerVolumeSpecName "kube-api-access-zhh6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:25.039534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.039495 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-token" pod "2829db12-be6a-46ba-a4d9-58158af0f3b4" (UID: "2829db12-be6a-46ba-a4d9-58158af0f3b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:25.039826 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.039799 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2829db12-be6a-46ba-a4d9-58158af0f3b4-istiod-ca-cert" pod "2829db12-be6a-46ba-a4d9-58158af0f3b4" (UID: "2829db12-be6a-46ba-a4d9-58158af0f3b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:25.039888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.039827 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-socket" pod "94bf0314-53b2-4937-9106-f72080e6d0f1" (UID: "94bf0314-53b2-4937-9106-f72080e6d0f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.040030 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.040011 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-certs" pod "94bf0314-53b2-4937-9106-f72080e6d0f1" (UID: "94bf0314-53b2-4937-9106-f72080e6d0f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.040107 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.040031 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-credential-socket" pod "94bf0314-53b2-4937-9106-f72080e6d0f1" (UID: "94bf0314-53b2-4937-9106-f72080e6d0f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.040176 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.040140 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-certs" pod "2829db12-be6a-46ba-a4d9-58158af0f3b4" (UID: "2829db12-be6a-46ba-a4d9-58158af0f3b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.041857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.040754 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-envoy" pod "4e0af82a-8a48-4758-97ca-48ddb633fda8" (UID: "4e0af82a-8a48-4758-97ca-48ddb633fda8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.041857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.040854 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-podinfo" pod "94bf0314-53b2-4937-9106-f72080e6d0f1" (UID: "94bf0314-53b2-4937-9106-f72080e6d0f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:25.041857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.040880 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-token" pod "4e0af82a-8a48-4758-97ca-48ddb633fda8" (UID: "4e0af82a-8a48-4758-97ca-48ddb633fda8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:25.041857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.041814 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-kube-api-access-kj58w" pod "94bf0314-53b2-4937-9106-f72080e6d0f1" (UID: "94bf0314-53b2-4937-9106-f72080e6d0f1"). InnerVolumeSpecName "kube-api-access-kj58w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:25.042188 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.042156 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-kube-api-access-hdvfg" pod "2829db12-be6a-46ba-a4d9-58158af0f3b4" (UID: "2829db12-be6a-46ba-a4d9-58158af0f3b4"). InnerVolumeSpecName "kube-api-access-hdvfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:25.042585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.042558 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-podinfo" pod "2829db12-be6a-46ba-a4d9-58158af0f3b4" (UID: "2829db12-be6a-46ba-a4d9-58158af0f3b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:25.043655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.043629 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-envoy" pod "2829db12-be6a-46ba-a4d9-58158af0f3b4" (UID: "2829db12-be6a-46ba-a4d9-58158af0f3b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.043922 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.043900 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-token" pod "94bf0314-53b2-4937-9106-f72080e6d0f1" (UID: "94bf0314-53b2-4937-9106-f72080e6d0f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:25.054546 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.054501 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:38:25.137290 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137254 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-certs\") pod \"08e1f58a-9117-4538-abe5-c8377b9f5434\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " Oct 31 21:38:25.137500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137302 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e1f58a-9117-4538-abe5-c8377b9f5434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e1f58a-9117-4538-abe5-c8377b9f5434-istiod-ca-cert\") pod \"08e1f58a-9117-4538-abe5-c8377b9f5434\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " Oct 31 21:38:25.137500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137376 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-kube-api-access-r2khw\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-kube-api-access-r2khw\") pod \"08e1f58a-9117-4538-abe5-c8377b9f5434\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " Oct 31 21:38:25.137500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137425 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-envoy\") pod \"08e1f58a-9117-4538-abe5-c8377b9f5434\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " Oct 31 21:38:25.137709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137500 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-data\") pod \"08e1f58a-9117-4538-abe5-c8377b9f5434\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " Oct 31 21:38:25.137709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137530 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-podinfo\") pod \"08e1f58a-9117-4538-abe5-c8377b9f5434\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " Oct 31 21:38:25.137709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137613 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-socket\") pod \"08e1f58a-9117-4538-abe5-c8377b9f5434\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " Oct 31 21:38:25.137709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137648 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-certs" pod "08e1f58a-9117-4538-abe5-c8377b9f5434" (UID: "08e1f58a-9117-4538-abe5-c8377b9f5434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.137932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137726 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-credential-socket\") pod \"08e1f58a-9117-4538-abe5-c8377b9f5434\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " Oct 31 21:38:25.137932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137756 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-token\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-token\") pod \"08e1f58a-9117-4538-abe5-c8377b9f5434\" (UID: \"08e1f58a-9117-4538-abe5-c8377b9f5434\") " Oct 31 21:38:25.137932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137862 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-data" pod "08e1f58a-9117-4538-abe5-c8377b9f5434" (UID: "08e1f58a-9117-4538-abe5-c8377b9f5434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.137932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.137902 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e1f58a-9117-4538-abe5-c8377b9f5434-istiod-ca-cert" pod "08e1f58a-9117-4538-abe5-c8377b9f5434" (UID: "08e1f58a-9117-4538-abe5-c8377b9f5434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:25.138483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138163 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-credential-socket" pod "08e1f58a-9117-4538-abe5-c8377b9f5434" (UID: "08e1f58a-9117-4538-abe5-c8377b9f5434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.138483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138220 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-socket" pod "08e1f58a-9117-4538-abe5-c8377b9f5434" (UID: "08e1f58a-9117-4538-abe5-c8377b9f5434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.138880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138854 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2829db12-be6a-46ba-a4d9-58158af0f3b4-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.138962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138889 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.138962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138904 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.138962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138920 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdvfg\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-kube-api-access-hdvfg\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.138962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138934 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj58w\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-kube-api-access-kj58w\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.138962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138952 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138966 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138980 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.138994 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08e1f58a-9117-4538-abe5-c8377b9f5434-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139007 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139020 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139036 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139048 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139060 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139072 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139083 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139096 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139108 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2829db12-be6a-46ba-a4d9-58158af0f3b4-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139120 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e0af82a-8a48-4758-97ca-48ddb633fda8-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139134 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139146 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139160 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e0af82a-8a48-4758-97ca-48ddb633fda8-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139173 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94bf0314-53b2-4937-9106-f72080e6d0f1-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139187 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e0af82a-8a48-4758-97ca-48ddb633fda8-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139202 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhh6k\" (UniqueName: \"kubernetes.io/projected/4e0af82a-8a48-4758-97ca-48ddb633fda8-kube-api-access-zhh6k\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.139216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139217 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94bf0314-53b2-4937-9106-f72080e6d0f1-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.140308 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.139231 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94bf0314-53b2-4937-9106-f72080e6d0f1-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.140481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.140447 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-kube-api-access-r2khw" pod "08e1f58a-9117-4538-abe5-c8377b9f5434" (UID: "08e1f58a-9117-4538-abe5-c8377b9f5434"). InnerVolumeSpecName "kube-api-access-r2khw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:25.140755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.140732 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-envoy" pod "08e1f58a-9117-4538-abe5-c8377b9f5434" (UID: "08e1f58a-9117-4538-abe5-c8377b9f5434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:25.140898 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.140873 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-podinfo" pod "08e1f58a-9117-4538-abe5-c8377b9f5434" (UID: "08e1f58a-9117-4538-abe5-c8377b9f5434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:25.142638 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.142614 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-token" pod "08e1f58a-9117-4538-abe5-c8377b9f5434" (UID: "08e1f58a-9117-4538-abe5-c8377b9f5434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:25.240366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.240316 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2khw\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-kube-api-access-r2khw\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.240366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.240364 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.240366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.240377 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.240627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.240386 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08e1f58a-9117-4538-abe5-c8377b9f5434-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:25.680837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.680787 5188 generic.go:356] "Generic (PLEG): container finished" podID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerID="962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd" exitCode=0 Oct 31 21:38:25.681315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.680889 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" Oct 31 21:38:25.681315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.680879 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" event={"ID":"4e0af82a-8a48-4758-97ca-48ddb633fda8","Type":"ContainerDied","Data":"962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd"} Oct 31 21:38:25.681315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.681074 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d" event={"ID":"4e0af82a-8a48-4758-97ca-48ddb633fda8","Type":"ContainerDied","Data":"7c1a51241a571f19859828ceafaee874b2fb263a94f9ef742f6a18d900c21113"} Oct 31 21:38:25.681315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.681125 5188 scope.go:122] "RemoveContainer" containerID="962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd" Oct 31 21:38:25.685113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.685077 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" Oct 31 21:38:25.685401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.685154 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c" event={"ID":"94bf0314-53b2-4937-9106-f72080e6d0f1","Type":"ContainerDied","Data":"1917d05ce64a107f5a750157d100476acc48cd33c56b0845847b8548527e8b06"} Oct 31 21:38:25.688423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.688390 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" event={"ID":"08e1f58a-9117-4538-abe5-c8377b9f5434","Type":"ContainerDied","Data":"da5270d1fe699f2d1a864437c102137e017c7fc26b9a9feebb12c22980cdcb3b"} Oct 31 21:38:25.689438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.689414 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk" Oct 31 21:38:25.700743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.700688 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" event={"ID":"2829db12-be6a-46ba-a4d9-58158af0f3b4","Type":"ContainerDied","Data":"f36311135e288c3def72d9c5333272a415abe27860033c65016151a1075de209"} Oct 31 21:38:25.700912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.700753 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96" Oct 31 21:38:25.701161 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.701135 5188 scope.go:122] "RemoveContainer" containerID="962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd" Oct 31 21:38:25.701693 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:38:25.701662 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd\": container with ID starting with 962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd not found: ID does not exist" containerID="962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd" Oct 31 21:38:25.701794 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.701704 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd"} err="failed to get container status \"962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd\": rpc error: code = NotFound desc = could not find container \"962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd\": container with ID starting with 962b5500b288185f0414f0a4e08f7cc177f12badcc04369b570a4d55bfaa70dd not found: ID does not exist" Oct 31 21:38:25.701794 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.701728 5188 scope.go:122] "RemoveContainer" containerID="9bd82551b7551f90adbb6e2a89f4bf5c7d87e4a8896ff02b536a05a333451832" Oct 31 21:38:25.716793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.716765 5188 scope.go:122] "RemoveContainer" containerID="a82dc61a344c5350fd08c6677630453f34b0fc3b086533ae1ddd257c75ebd116" Oct 31 21:38:25.729548 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.729519 5188 scope.go:122] "RemoveContainer" containerID="0e36e3cc2c63c78607c6637f75b461a368c97cfc1d52e518b199d19647a6cfc6" Oct 31 21:38:25.757047 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.757010 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d"] Oct 31 21:38:25.766654 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.766619 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-xkf9d"] Oct 31 21:38:25.797289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.797242 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96"] Oct 31 21:38:25.801274 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.801220 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rvl96"] Oct 31 21:38:25.829080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.829042 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk"] Oct 31 21:38:25.837572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.837538 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-rg6jk"] Oct 31 21:38:25.868437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.868397 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c"] Oct 31 21:38:25.879146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:25.879106 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-gms8c"] Oct 31 21:38:26.704190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.704155 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9"] Oct 31 21:38:26.705067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705042 5188 cpu_manager.go:414] "containerMap: removing container" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705068 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705109 5188 cpu_manager.go:414] "containerMap: removing container" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705118 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705132 5188 cpu_manager.go:414] "containerMap: removing container" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705139 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705158 5188 cpu_manager.go:414] "containerMap: removing container" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705166 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705181 5188 cpu_manager.go:414] "containerMap: removing container" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705187 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" Oct 31 21:38:26.705204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705205 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705212 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705228 5188 cpu_manager.go:414] "containerMap: removing container" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705237 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705414 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a17c3f-4dc9-4d95-9e63-1ed55bdb363b" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705433 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705446 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705460 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705474 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705486 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705495 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="776d7137-587e-4c9a-8fbb-648a39b77e34" containerName="istio-proxy" Oct 31 21:38:26.705697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.705505 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c36232b3-9979-41d1-97fe-518e3a25ac76" containerName="istio-proxy" Oct 31 21:38:26.711079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.711042 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.753218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.753165 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:38:26.753466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.753345 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:38:26.753466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.753445 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-07afdd-serving-cert\"" Oct 31 21:38:26.761081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.761042 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9"] Oct 31 21:38:26.762756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.762719 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74313ead-925b-4cc6-90f8-b91fff13c1c1-openshift-service-ca\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.762919 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.762772 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-metrics-certs\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.762919 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.762880 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-maas-api-tls\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.762919 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.762907 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6k7s\" (UniqueName: \"kubernetes.io/projected/74313ead-925b-4cc6-90f8-b91fff13c1c1-kube-api-access-p6k7s\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.763084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.762957 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 21:38:26.763084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.763069 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kmnpr\"" Oct 31 21:38:26.864213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.864165 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74313ead-925b-4cc6-90f8-b91fff13c1c1-openshift-service-ca\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.864213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.864220 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-metrics-certs\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.864540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.864267 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-maas-api-tls\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.864540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.864289 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6k7s\" (UniqueName: \"kubernetes.io/projected/74313ead-925b-4cc6-90f8-b91fff13c1c1-kube-api-access-p6k7s\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.865054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.865029 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74313ead-925b-4cc6-90f8-b91fff13c1c1-openshift-service-ca\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.867587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.867546 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-metrics-certs\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.867742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.867718 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-maas-api-tls\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:26.888274 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:26.888230 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6k7s\" (UniqueName: \"kubernetes.io/projected/74313ead-925b-4cc6-90f8-b91fff13c1c1-kube-api-access-p6k7s\") pod \"maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:27.028583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:27.028491 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:27.360261 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:27.360221 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9"] Oct 31 21:38:27.360655 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:27.360631 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74313ead_925b_4cc6_90f8_b91fff13c1c1.slice/crio-09a93118c1ec50771ee9a65cd2a06728c1eed8a263018b2fe9db0c08459898c1 WatchSource:0}: Error finding container 09a93118c1ec50771ee9a65cd2a06728c1eed8a263018b2fe9db0c08459898c1: Status 404 returned error can't find the container with id 09a93118c1ec50771ee9a65cd2a06728c1eed8a263018b2fe9db0c08459898c1 Oct 31 21:38:27.474654 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:27.474613 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e1f58a-9117-4538-abe5-c8377b9f5434" path="/var/lib/kubelet/pods/08e1f58a-9117-4538-abe5-c8377b9f5434/volumes" Oct 31 21:38:27.475162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:27.475129 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" path="/var/lib/kubelet/pods/2829db12-be6a-46ba-a4d9-58158af0f3b4/volumes" Oct 31 21:38:27.475635 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:27.475617 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e0af82a-8a48-4758-97ca-48ddb633fda8" path="/var/lib/kubelet/pods/4e0af82a-8a48-4758-97ca-48ddb633fda8/volumes" Oct 31 21:38:27.476514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:27.476493 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94bf0314-53b2-4937-9106-f72080e6d0f1" path="/var/lib/kubelet/pods/94bf0314-53b2-4937-9106-f72080e6d0f1/volumes" Oct 31 21:38:27.730531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:27.730486 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" event={"ID":"74313ead-925b-4cc6-90f8-b91fff13c1c1","Type":"ContainerStarted","Data":"09a93118c1ec50771ee9a65cd2a06728c1eed8a263018b2fe9db0c08459898c1"} Oct 31 21:38:29.314353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:29.313857 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc"] Oct 31 21:38:29.315124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:29.315092 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97"] Oct 31 21:38:29.315631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:29.315596 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" containerID="cri-o://8c5a2e32ab7fecce2f0c77b7e138903284415a378fe287efdde519143890a025" gracePeriod=30 Oct 31 21:38:29.316175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:29.316130 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" containerID="cri-o://2ff6847a9ebe2a567252267affd167e40c7f7b4376392f88cfa1e0d585beb0ca" gracePeriod=30 Oct 31 21:38:29.335854 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:29.331342 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh"] Oct 31 21:38:29.335854 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:29.331766 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" containerID="cri-o://d05cd36aa43ad319cbfc62c76cf8f88fba35885bddfe6db359012049ea262f9c" gracePeriod=30 Oct 31 21:38:29.762314 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:29.762255 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" event={"ID":"74313ead-925b-4cc6-90f8-b91fff13c1c1","Type":"ContainerStarted","Data":"2cb3cf04151f65ca9d00155bea7e500e33224300ed37c943a1b3441fd26cc050"} Oct 31 21:38:29.762591 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:29.762392 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:29.802592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:29.802529 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" podStartSLOduration=2.1260562849999998 podStartE2EDuration="3.802511457s" totalImagesPullingTime="1.676455172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:29.80103709 +0000 UTC m=+1356.493994484" watchObservedRunningTime="2026-10-31 21:38:29.802511457 +0000 UTC m=+1356.495468850" Oct 31 21:38:32.370830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.366724 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p"] Oct 31 21:38:32.370830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.367663 5188 cpu_manager.go:414] "containerMap: removing container" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" Oct 31 21:38:32.370830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.367684 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2829db12-be6a-46ba-a4d9-58158af0f3b4" containerName="istio-proxy" Oct 31 21:38:32.381971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.381934 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:32.390010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.389891 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p"] Oct 31 21:38:32.411757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.411718 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fc0d65-dockercfg-rx4fq\"" Oct 31 21:38:32.411967 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.411930 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fc0d65\"" Oct 31 21:38:32.497687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.497651 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qlvd\" (UniqueName: \"kubernetes.io/projected/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-kube-api-access-5qlvd\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p\" (UID: \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:32.497891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.497864 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p\" (UID: \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:32.521992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.521953 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt"] Oct 31 21:38:32.526537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.526506 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:32.579367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.572846 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt"] Oct 31 21:38:32.600800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.600763 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2104e85-d811-4b12-9628-226615b41f16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt\" (UID: \"d2104e85-d811-4b12-9628-226615b41f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:32.601052 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.601038 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p\" (UID: \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:32.601151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.601139 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz4n4\" (UniqueName: \"kubernetes.io/projected/d2104e85-d811-4b12-9628-226615b41f16-kube-api-access-wz4n4\") pod \"payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt\" (UID: \"d2104e85-d811-4b12-9628-226615b41f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:32.601661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.601534 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qlvd\" (UniqueName: \"kubernetes.io/projected/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-kube-api-access-5qlvd\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p\" (UID: \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:32.603069 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.603045 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p\" (UID: \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:32.615629 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:38:32.615580 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:38:32.628152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.628072 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qlvd\" (UniqueName: \"kubernetes.io/projected/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-kube-api-access-5qlvd\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p\" (UID: \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:32.634438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.632980 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:38:32.703375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.703311 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2104e85-d811-4b12-9628-226615b41f16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt\" (UID: \"d2104e85-d811-4b12-9628-226615b41f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:32.703574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.703426 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz4n4\" (UniqueName: \"kubernetes.io/projected/d2104e85-d811-4b12-9628-226615b41f16-kube-api-access-wz4n4\") pod \"payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt\" (UID: \"d2104e85-d811-4b12-9628-226615b41f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:32.704377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.704341 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2104e85-d811-4b12-9628-226615b41f16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt\" (UID: \"d2104e85-d811-4b12-9628-226615b41f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:32.713005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.712958 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:32.736970 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.736930 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz4n4\" (UniqueName: \"kubernetes.io/projected/d2104e85-d811-4b12-9628-226615b41f16-kube-api-access-wz4n4\") pod \"payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt\" (UID: \"d2104e85-d811-4b12-9628-226615b41f16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:32.850587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:32.850550 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:33.056724 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:33.056683 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2efa7d_10e7_4ac2_a9ac_44f96f561445.slice/crio-d443de8282233048bb26b18707dc6f820b241e1a1a6b6341f984ecd65379d554 WatchSource:0}: Error finding container d443de8282233048bb26b18707dc6f820b241e1a1a6b6341f984ecd65379d554: Status 404 returned error can't find the container with id d443de8282233048bb26b18707dc6f820b241e1a1a6b6341f984ecd65379d554 Oct 31 21:38:33.056864 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:33.056723 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p"] Oct 31 21:38:33.234586 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:33.233174 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt"] Oct 31 21:38:33.645221 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:33.645184 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:38:33.823521 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:33.823472 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" event={"ID":"d2104e85-d811-4b12-9628-226615b41f16","Type":"ContainerStarted","Data":"f7838c4573ed8082f5e2b2af1bd7e6a0a8739fbeb5a431fe890578ccf085aef4"} Oct 31 21:38:33.825635 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:33.825594 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" event={"ID":"bd2efa7d-10e7-4ac2-a9ac-44f96f561445","Type":"ContainerStarted","Data":"d443de8282233048bb26b18707dc6f820b241e1a1a6b6341f984ecd65379d554"} Oct 31 21:38:34.860385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:34.860123 5188 generic.go:356] "Generic (PLEG): container finished" podID="420fbd0f-3194-49cb-9a43-191992684189" containerID="8c5a2e32ab7fecce2f0c77b7e138903284415a378fe287efdde519143890a025" exitCode=0 Oct 31 21:38:34.860861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:34.860828 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" event={"ID":"420fbd0f-3194-49cb-9a43-191992684189","Type":"ContainerDied","Data":"8c5a2e32ab7fecce2f0c77b7e138903284415a378fe287efdde519143890a025"} Oct 31 21:38:34.887570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:34.887409 5188 generic.go:356] "Generic (PLEG): container finished" podID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerID="d05cd36aa43ad319cbfc62c76cf8f88fba35885bddfe6db359012049ea262f9c" exitCode=0 Oct 31 21:38:34.887700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:34.887631 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" event={"ID":"0f4bd232-cac5-4ab4-aedb-5242b6e59f81","Type":"ContainerDied","Data":"d05cd36aa43ad319cbfc62c76cf8f88fba35885bddfe6db359012049ea262f9c"} Oct 31 21:38:34.889641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:34.889610 5188 generic.go:356] "Generic (PLEG): container finished" podID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerID="2ff6847a9ebe2a567252267affd167e40c7f7b4376392f88cfa1e0d585beb0ca" exitCode=0 Oct 31 21:38:34.889804 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:34.889786 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" event={"ID":"bf81bcb4-599c-4aca-87ab-64f964cf6e54","Type":"ContainerDied","Data":"2ff6847a9ebe2a567252267affd167e40c7f7b4376392f88cfa1e0d585beb0ca"} Oct 31 21:38:34.931730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:34.931701 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:38:35.057207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.057175 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:38:35.062002 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.061973 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:38:35.086010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.085973 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-podinfo\") pod \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " Oct 31 21:38:35.086186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.086112 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-certs\") pod \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " Oct 31 21:38:35.086186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.086167 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-credential-socket\") pod \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " Oct 31 21:38:35.086337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.086195 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istiod-ca-cert\") pod \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " Oct 31 21:38:35.086337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.086241 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-socket\") pod \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " Oct 31 21:38:35.086337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.086290 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-envoy\") pod \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " Oct 31 21:38:35.086504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.086338 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-token\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-token\") pod \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " Oct 31 21:38:35.086504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.086368 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-data\") pod \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " Oct 31 21:38:35.086504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.086418 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-kube-api-access-c2d8g\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-kube-api-access-c2d8g\") pod \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\" (UID: \"bf81bcb4-599c-4aca-87ab-64f964cf6e54\") " Oct 31 21:38:35.088732 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.088680 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-credential-socket" pod "bf81bcb4-599c-4aca-87ab-64f964cf6e54" (UID: "bf81bcb4-599c-4aca-87ab-64f964cf6e54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.088961 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.088931 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-certs" pod "bf81bcb4-599c-4aca-87ab-64f964cf6e54" (UID: "bf81bcb4-599c-4aca-87ab-64f964cf6e54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.089218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.089201 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-socket" pod "bf81bcb4-599c-4aca-87ab-64f964cf6e54" (UID: "bf81bcb4-599c-4aca-87ab-64f964cf6e54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.089550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.089529 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-data" pod "bf81bcb4-599c-4aca-87ab-64f964cf6e54" (UID: "bf81bcb4-599c-4aca-87ab-64f964cf6e54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.090178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.090156 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istiod-ca-cert" pod "bf81bcb4-599c-4aca-87ab-64f964cf6e54" (UID: "bf81bcb4-599c-4aca-87ab-64f964cf6e54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:35.092531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.092504 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-kube-api-access-c2d8g" pod "bf81bcb4-599c-4aca-87ab-64f964cf6e54" (UID: "bf81bcb4-599c-4aca-87ab-64f964cf6e54"). InnerVolumeSpecName "kube-api-access-c2d8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:35.095259 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.095229 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-token" pod "bf81bcb4-599c-4aca-87ab-64f964cf6e54" (UID: "bf81bcb4-599c-4aca-87ab-64f964cf6e54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:35.097976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.097946 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-envoy" pod "bf81bcb4-599c-4aca-87ab-64f964cf6e54" (UID: "bf81bcb4-599c-4aca-87ab-64f964cf6e54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.098207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.098186 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-podinfo" pod "bf81bcb4-599c-4aca-87ab-64f964cf6e54" (UID: "bf81bcb4-599c-4aca-87ab-64f964cf6e54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:35.203187 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.203059 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-socket\") pod \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " Oct 31 21:38:35.203722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.203703 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-kube-api-access-hjhbl\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-kube-api-access-hjhbl\") pod \"420fbd0f-3194-49cb-9a43-191992684189\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " Oct 31 21:38:35.203914 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.203895 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-data\") pod \"420fbd0f-3194-49cb-9a43-191992684189\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " Oct 31 21:38:35.204027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.204014 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/420fbd0f-3194-49cb-9a43-191992684189-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420fbd0f-3194-49cb-9a43-191992684189-istio-podinfo\") pod \"420fbd0f-3194-49cb-9a43-191992684189\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " Oct 31 21:38:35.204159 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.204147 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-data\") pod \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " Oct 31 21:38:35.204409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.204390 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-istio-token\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-istio-token\") pod \"420fbd0f-3194-49cb-9a43-191992684189\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " Oct 31 21:38:35.204532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.204518 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-credential-socket\") pod \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " Oct 31 21:38:35.206248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.206227 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-envoy\") pod \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " Oct 31 21:38:35.206781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.206763 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-certs\") pod \"420fbd0f-3194-49cb-9a43-191992684189\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " Oct 31 21:38:35.207154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.207138 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istiod-ca-cert\") pod \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " Oct 31 21:38:35.207857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.207838 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-token\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-token\") pod \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " Oct 31 21:38:35.208115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.208100 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-socket\") pod \"420fbd0f-3194-49cb-9a43-191992684189\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " Oct 31 21:38:35.208609 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.208592 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/420fbd0f-3194-49cb-9a43-191992684189-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420fbd0f-3194-49cb-9a43-191992684189-istiod-ca-cert\") pod \"420fbd0f-3194-49cb-9a43-191992684189\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " Oct 31 21:38:35.209128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.209108 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-credential-socket\") pod \"420fbd0f-3194-49cb-9a43-191992684189\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " Oct 31 21:38:35.209755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.209737 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-kube-api-access-sswbb\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-kube-api-access-sswbb\") pod \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " Oct 31 21:38:35.210243 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.210225 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-podinfo\") pod \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " Oct 31 21:38:35.210795 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.203648 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-socket" pod "0f4bd232-cac5-4ab4-aedb-5242b6e59f81" (UID: "0f4bd232-cac5-4ab4-aedb-5242b6e59f81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.205694 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-data" pod "0f4bd232-cac5-4ab4-aedb-5242b6e59f81" (UID: "0f4bd232-cac5-4ab4-aedb-5242b6e59f81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.205982 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-credential-socket" pod "0f4bd232-cac5-4ab4-aedb-5242b6e59f81" (UID: "0f4bd232-cac5-4ab4-aedb-5242b6e59f81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.206682 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-data" pod "420fbd0f-3194-49cb-9a43-191992684189" (UID: "420fbd0f-3194-49cb-9a43-191992684189"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.207072 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-certs" pod "420fbd0f-3194-49cb-9a43-191992684189" (UID: "420fbd0f-3194-49cb-9a43-191992684189"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.207705 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istiod-ca-cert" pod "0f4bd232-cac5-4ab4-aedb-5242b6e59f81" (UID: "0f4bd232-cac5-4ab4-aedb-5242b6e59f81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.208035 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-istio-token" pod "420fbd0f-3194-49cb-9a43-191992684189" (UID: "420fbd0f-3194-49cb-9a43-191992684189"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.208518 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-socket" pod "420fbd0f-3194-49cb-9a43-191992684189" (UID: "420fbd0f-3194-49cb-9a43-191992684189"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.209053 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420fbd0f-3194-49cb-9a43-191992684189-istiod-ca-cert" pod "420fbd0f-3194-49cb-9a43-191992684189" (UID: "420fbd0f-3194-49cb-9a43-191992684189"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.209527 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-credential-socket" pod "420fbd0f-3194-49cb-9a43-191992684189" (UID: "420fbd0f-3194-49cb-9a43-191992684189"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.211027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.209613 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/420fbd0f-3194-49cb-9a43-191992684189-istio-podinfo" pod "420fbd0f-3194-49cb-9a43-191992684189" (UID: "420fbd0f-3194-49cb-9a43-191992684189"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:35.211866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.211829 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-token" pod "0f4bd232-cac5-4ab4-aedb-5242b6e59f81" (UID: "0f4bd232-cac5-4ab4-aedb-5242b6e59f81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:35.212027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.212012 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-envoy\") pod \"420fbd0f-3194-49cb-9a43-191992684189\" (UID: \"420fbd0f-3194-49cb-9a43-191992684189\") " Oct 31 21:38:35.212691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.212674 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-certs\") pod \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\" (UID: \"0f4bd232-cac5-4ab4-aedb-5242b6e59f81\") " Oct 31 21:38:35.213245 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.213083 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-certs" pod "0f4bd232-cac5-4ab4-aedb-5242b6e59f81" (UID: "0f4bd232-cac5-4ab4-aedb-5242b6e59f81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.214035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214012 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214134 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214215 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214315 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214305 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214439 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420fbd0f-3194-49cb-9a43-191992684189-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214536 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214525 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214601 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214675 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214747 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214887 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214875 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.214965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.214955 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215047 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215036 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215118 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215109 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215186 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420fbd0f-3194-49cb-9a43-191992684189-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215264 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215366 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215457 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215598 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf81bcb4-599c-4aca-87ab-64f964cf6e54-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215766 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215881 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.215976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.215966 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.216058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.216047 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2d8g\" (UniqueName: \"kubernetes.io/projected/bf81bcb4-599c-4aca-87ab-64f964cf6e54-kube-api-access-c2d8g\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.217042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.217001 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-kube-api-access-sswbb" pod "0f4bd232-cac5-4ab4-aedb-5242b6e59f81" (UID: "0f4bd232-cac5-4ab4-aedb-5242b6e59f81"). InnerVolumeSpecName "kube-api-access-sswbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:35.217843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.217780 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-envoy" pod "420fbd0f-3194-49cb-9a43-191992684189" (UID: "420fbd0f-3194-49cb-9a43-191992684189"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.227247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.227147 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-kube-api-access-hjhbl" pod "420fbd0f-3194-49cb-9a43-191992684189" (UID: "420fbd0f-3194-49cb-9a43-191992684189"). InnerVolumeSpecName "kube-api-access-hjhbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:35.228935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.228867 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-podinfo" pod "0f4bd232-cac5-4ab4-aedb-5242b6e59f81" (UID: "0f4bd232-cac5-4ab4-aedb-5242b6e59f81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:35.230245 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.230124 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-envoy" pod "0f4bd232-cac5-4ab4-aedb-5242b6e59f81" (UID: "0f4bd232-cac5-4ab4-aedb-5242b6e59f81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:35.317781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.317735 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sswbb\" (UniqueName: \"kubernetes.io/projected/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-kube-api-access-sswbb\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.317781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.317782 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.318020 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.317798 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420fbd0f-3194-49cb-9a43-191992684189-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.318020 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.317812 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjhbl\" (UniqueName: \"kubernetes.io/projected/420fbd0f-3194-49cb-9a43-191992684189-kube-api-access-hjhbl\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.318020 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.317825 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f4bd232-cac5-4ab4-aedb-5242b6e59f81-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:38:35.784801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.784706 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:38:35.923362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.923279 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" event={"ID":"bd2efa7d-10e7-4ac2-a9ac-44f96f561445","Type":"ContainerStarted","Data":"17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a"} Oct 31 21:38:35.924081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.924059 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:35.925539 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.925523 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" Oct 31 21:38:35.925739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.925568 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97" event={"ID":"420fbd0f-3194-49cb-9a43-191992684189","Type":"ContainerDied","Data":"89d3a95898452abc320dd7b9d45e31c99696ffafc6e2e56ede415a8ecc8b9413"} Oct 31 21:38:35.925813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.925772 5188 scope.go:122] "RemoveContainer" containerID="8c5a2e32ab7fecce2f0c77b7e138903284415a378fe287efdde519143890a025" Oct 31 21:38:35.930415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.929772 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" event={"ID":"d2104e85-d811-4b12-9628-226615b41f16","Type":"ContainerStarted","Data":"5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589"} Oct 31 21:38:35.930415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.929832 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:35.932510 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.932481 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" event={"ID":"0f4bd232-cac5-4ab4-aedb-5242b6e59f81","Type":"ContainerDied","Data":"4fbef5455c5fadc9b106e2ecba28c3bb2c1fe16bd6430a327ffcfbcc6c4f89c7"} Oct 31 21:38:35.932604 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.932589 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh" Oct 31 21:38:35.936566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.936544 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" Oct 31 21:38:35.936763 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.936562 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc" event={"ID":"bf81bcb4-599c-4aca-87ab-64f964cf6e54","Type":"ContainerDied","Data":"4eb13a17757036c61446a7e1593008c6d890f235e09bd8caafdf94563137c48f"} Oct 31 21:38:35.952632 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.952564 5188 scope.go:122] "RemoveContainer" containerID="d05cd36aa43ad319cbfc62c76cf8f88fba35885bddfe6db359012049ea262f9c" Oct 31 21:38:35.966075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.966018 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" podStartSLOduration=2.17312102 podStartE2EDuration="3.96600017s" totalImagesPullingTime="1.79287915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:35.961821081 +0000 UTC m=+1362.654778473" watchObservedRunningTime="2026-10-31 21:38:35.96600017 +0000 UTC m=+1362.658957565" Oct 31 21:38:35.971745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:35.971018 5188 scope.go:122] "RemoveContainer" containerID="2ff6847a9ebe2a567252267affd167e40c7f7b4376392f88cfa1e0d585beb0ca" Oct 31 21:38:36.055640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.055510 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" podStartSLOduration=2.445375153 podStartE2EDuration="4.055491824s" totalImagesPullingTime="1.610116671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:36.007568065 +0000 UTC m=+1362.700525459" watchObservedRunningTime="2026-10-31 21:38:36.055491824 +0000 UTC m=+1362.748449218" Oct 31 21:38:36.061391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.061353 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc"] Oct 31 21:38:36.072909 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.072862 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-h47qc"] Oct 31 21:38:36.123841 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.123800 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh"] Oct 31 21:38:36.130394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.130362 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-k8mwh"] Oct 31 21:38:36.177051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.177003 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97"] Oct 31 21:38:36.184924 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.184875 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-knt97"] Oct 31 21:38:36.232350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.232296 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q"] Oct 31 21:38:36.233258 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.233240 5188 cpu_manager.go:414] "containerMap: removing container" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" Oct 31 21:38:36.233302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.233261 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" Oct 31 21:38:36.233302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.233286 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" Oct 31 21:38:36.233302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.233294 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" Oct 31 21:38:36.233441 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.233333 5188 cpu_manager.go:414] "containerMap: removing container" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" Oct 31 21:38:36.233441 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.233342 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" Oct 31 21:38:36.233503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.233481 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" containerName="istio-proxy" Oct 31 21:38:36.233503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.233500 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="420fbd0f-3194-49cb-9a43-191992684189" containerName="istio-proxy" Oct 31 21:38:36.233568 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.233517 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" containerName="istio-proxy" Oct 31 21:38:36.239588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.239557 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.253244 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.253212 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c744c0\"/\"kube-root-ca.crt\"" Oct 31 21:38:36.253668 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.253638 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c744c0\"/\"openshift-service-ca.crt\"" Oct 31 21:38:36.253776 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.253688 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c744c0\"/\"tenantcfg-model-c744c0-kserve-self-signed-certs\"" Oct 31 21:38:36.253857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.253829 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c744c0\"/\"default-dockercfg-xcxlq\"" Oct 31 21:38:36.256215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.256187 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q"] Oct 31 21:38:36.330450 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.330403 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-home\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.330629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.330540 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-dshm\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.330726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.330662 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2610999-7b04-479d-9322-fd0dadd6d0ea-tls-certs\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.330788 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.330721 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-model-cache\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.330940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.330916 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-tmp-dir\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.331079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.330964 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ngt\" (UniqueName: \"kubernetes.io/projected/f2610999-7b04-479d-9322-fd0dadd6d0ea-kube-api-access-l8ngt\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.432005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.431965 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-tmp-dir\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.432189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.432014 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ngt\" (UniqueName: \"kubernetes.io/projected/f2610999-7b04-479d-9322-fd0dadd6d0ea-kube-api-access-l8ngt\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.432189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.432050 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-home\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.432189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.432117 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-dshm\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.432189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.432176 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2610999-7b04-479d-9322-fd0dadd6d0ea-tls-certs\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.432396 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.432210 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-model-cache\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.432457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.432432 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-tmp-dir\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.432668 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.432646 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-model-cache\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.432732 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.432708 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-home\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.434754 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.434727 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-dshm\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.434918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.434900 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2610999-7b04-479d-9322-fd0dadd6d0ea-tls-certs\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.454380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.454310 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ngt\" (UniqueName: \"kubernetes.io/projected/f2610999-7b04-479d-9322-fd0dadd6d0ea-kube-api-access-l8ngt\") pod \"tenantcfg-model-c744c0-kserve-dcdfc486f-7676q\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.562070 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.562032 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:36.967431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:36.967350 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q"] Oct 31 21:38:36.969083 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:36.969028 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2610999_7b04_479d_9322_fd0dadd6d0ea.slice/crio-d6475fbe32c19ac3123559257e10b62e88066d50578861fc15ce5a2f456106d7 WatchSource:0}: Error finding container d6475fbe32c19ac3123559257e10b62e88066d50578861fc15ce5a2f456106d7: Status 404 returned error can't find the container with id d6475fbe32c19ac3123559257e10b62e88066d50578861fc15ce5a2f456106d7 Oct 31 21:38:37.476863 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:37.476813 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4bd232-cac5-4ab4-aedb-5242b6e59f81" path="/var/lib/kubelet/pods/0f4bd232-cac5-4ab4-aedb-5242b6e59f81/volumes" Oct 31 21:38:37.477548 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:37.477528 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="420fbd0f-3194-49cb-9a43-191992684189" path="/var/lib/kubelet/pods/420fbd0f-3194-49cb-9a43-191992684189/volumes" Oct 31 21:38:37.478174 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:37.478155 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf81bcb4-599c-4aca-87ab-64f964cf6e54" path="/var/lib/kubelet/pods/bf81bcb4-599c-4aca-87ab-64f964cf6e54/volumes" Oct 31 21:38:37.962548 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:37.962501 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" event={"ID":"f2610999-7b04-479d-9322-fd0dadd6d0ea","Type":"ContainerStarted","Data":"b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf"} Oct 31 21:38:37.962777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:37.962560 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" event={"ID":"f2610999-7b04-479d-9322-fd0dadd6d0ea","Type":"ContainerStarted","Data":"d6475fbe32c19ac3123559257e10b62e88066d50578861fc15ce5a2f456106d7"} Oct 31 21:38:38.008077 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:38.008018 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" podStartSLOduration=2.008003139 podStartE2EDuration="2.008003139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:38.004761676 +0000 UTC m=+1364.697719072" watchObservedRunningTime="2026-10-31 21:38:38.008003139 +0000 UTC m=+1364.700960531" Oct 31 21:38:38.562708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:38.562662 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:40.681191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.681153 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn"] Oct 31 21:38:40.687076 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.687051 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.718351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.715657 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn"] Oct 31 21:38:40.793882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.793843 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-metrics-certs\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.794058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.793996 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a00a2fc-729d-4fcb-961b-812f90b42430-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.794104 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.794096 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.794140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.794126 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvjv\" (UniqueName: \"kubernetes.io/projected/7a00a2fc-729d-4fcb-961b-812f90b42430-kube-api-access-ktvjv\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.801544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.801519 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d75c94-serving-cert\"" Oct 31 21:38:40.895694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.895645 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-metrics-certs\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.895892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.895729 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a00a2fc-729d-4fcb-961b-812f90b42430-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.895892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.895800 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.895892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.895821 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvjv\" (UniqueName: \"kubernetes.io/projected/7a00a2fc-729d-4fcb-961b-812f90b42430-kube-api-access-ktvjv\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.896631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.896607 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a00a2fc-729d-4fcb-961b-812f90b42430-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.898430 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.898392 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.898538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.898433 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-metrics-certs\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:40.928846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:40.928787 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvjv\" (UniqueName: \"kubernetes.io/projected/7a00a2fc-729d-4fcb-961b-812f90b42430-kube-api-access-ktvjv\") pod \"maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:41.003729 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.003635 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:41.208568 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.208517 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c"] Oct 31 21:38:41.215117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.215083 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.241174 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.241141 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-752623-openshift-default-dockercfg-plmjm\"" Oct 31 21:38:41.245793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.245758 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c"] Oct 31 21:38:41.346443 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.346409 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn"] Oct 31 21:38:41.347731 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:41.347692 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a00a2fc_729d_4fcb_961b_812f90b42430.slice/crio-7aabaeed605237feff5d35f73c8e8c9381d4450b407e9576f9a3e7e02cd696f5 WatchSource:0}: Error finding container 7aabaeed605237feff5d35f73c8e8c9381d4450b407e9576f9a3e7e02cd696f5: Status 404 returned error can't find the container with id 7aabaeed605237feff5d35f73c8e8c9381d4450b407e9576f9a3e7e02cd696f5 Oct 31 21:38:41.412346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.412139 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-socket\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.412346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.412245 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-data\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.412346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.412282 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-envoy\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.412709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.412349 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmpd\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-kube-api-access-kcmpd\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.412709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.412433 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istiod-ca-cert\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.412709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.412668 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-token\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.435350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.434514 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-certs\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.435350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.434613 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-credential-socket\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.435350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.434683 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-podinfo\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.536314 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.536268 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-credential-socket\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.536547 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.536420 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-podinfo\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.536789 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.536761 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-credential-socket\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.537093 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.537074 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-socket\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.537390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.536489 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-socket\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.537529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.537496 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-data\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.537600 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.537537 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-envoy\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.537600 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.537579 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmpd\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-kube-api-access-kcmpd\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.537705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.537610 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istiod-ca-cert\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.537764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.537700 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-token\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.537816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.537768 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-certs\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.538180 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.538155 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-certs\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.540063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.540005 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istiod-ca-cert\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.540486 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.540463 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-data\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.541235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.541211 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-podinfo\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.542615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.542590 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-envoy\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.561070 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.561039 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmpd\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-kube-api-access-kcmpd\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.561554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.561534 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-token\") pod \"e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:41.834169 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:41.834145 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:42.015687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:42.015584 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" event={"ID":"7a00a2fc-729d-4fcb-961b-812f90b42430","Type":"ContainerStarted","Data":"a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803"} Oct 31 21:38:42.015687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:42.015653 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" event={"ID":"7a00a2fc-729d-4fcb-961b-812f90b42430","Type":"ContainerStarted","Data":"7aabaeed605237feff5d35f73c8e8c9381d4450b407e9576f9a3e7e02cd696f5"} Oct 31 21:38:42.015687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:42.015682 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:42.081476 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:42.081413 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" podStartSLOduration=1.6714091 podStartE2EDuration="2.081392718s" totalImagesPullingTime="409.983618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:42.078130703 +0000 UTC m=+1368.771088099" watchObservedRunningTime="2026-10-31 21:38:42.081392718 +0000 UTC m=+1368.774350112" Oct 31 21:38:42.184702 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:42.184667 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c"] Oct 31 21:38:42.185451 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:42.185428 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6255c981_759d_4a87_b6a1_dfb15b19fd6c.slice/crio-cd9f6e72446081933dbc742ce6f5a927c0aec254cd14c1c6dae42b7895a6c86d WatchSource:0}: Error finding container cd9f6e72446081933dbc742ce6f5a927c0aec254cd14c1c6dae42b7895a6c86d: Status 404 returned error can't find the container with id cd9f6e72446081933dbc742ce6f5a927c0aec254cd14c1c6dae42b7895a6c86d Oct 31 21:38:42.188213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:42.188175 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:38:42.188299 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:42.188264 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:38:42.188369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:42.188307 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:38:43.026995 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:43.026951 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" event={"ID":"6255c981-759d-4a87-b6a1-dfb15b19fd6c","Type":"ContainerStarted","Data":"e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe"} Oct 31 21:38:43.027464 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:43.027014 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" event={"ID":"6255c981-759d-4a87-b6a1-dfb15b19fd6c","Type":"ContainerStarted","Data":"cd9f6e72446081933dbc742ce6f5a927c0aec254cd14c1c6dae42b7895a6c86d"} Oct 31 21:38:43.080348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:43.080279 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" podStartSLOduration=2.080263173 podStartE2EDuration="2.080263173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:43.070507372 +0000 UTC m=+1369.763464766" watchObservedRunningTime="2026-10-31 21:38:43.080263173 +0000 UTC m=+1369.773220565" Oct 31 21:38:43.835434 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:43.835394 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:43.842592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:43.842559 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:44.036138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:44.036103 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:44.037378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:44.037312 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:38:46.563133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:46.563092 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:46.577107 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:46.577071 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:46.956112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:46.956077 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:38:47.080724 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.080695 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:38:47.579575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.579534 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls"] Oct 31 21:38:47.585190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.585156 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:38:47.610379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.610344 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-752623\"" Oct 31 21:38:47.610612 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.610525 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-752623-dockercfg-tk8hl\"" Oct 31 21:38:47.614435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.614404 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls"] Oct 31 21:38:47.669193 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.669138 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w"] Oct 31 21:38:47.674715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.674685 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:38:47.704873 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.703336 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w"] Oct 31 21:38:47.747349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.746532 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjk9m\" (UniqueName: \"kubernetes.io/projected/e36c8273-fbb4-45dc-bec2-087f65da1f46-kube-api-access-hjk9m\") pod \"payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls\" (UID: \"e36c8273-fbb4-45dc-bec2-087f65da1f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:38:47.747349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.746809 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e36c8273-fbb4-45dc-bec2-087f65da1f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls\" (UID: \"e36c8273-fbb4-45dc-bec2-087f65da1f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:38:47.847916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.847820 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82svc\" (UniqueName: \"kubernetes.io/projected/bc4521b5-5894-44dc-9419-8cb97d8dcba1-kube-api-access-82svc\") pod \"payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w\" (UID: \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:38:47.848099 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.847951 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4521b5-5894-44dc-9419-8cb97d8dcba1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w\" (UID: \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:38:47.848099 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.848042 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e36c8273-fbb4-45dc-bec2-087f65da1f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls\" (UID: \"e36c8273-fbb4-45dc-bec2-087f65da1f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:38:47.848217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.848154 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjk9m\" (UniqueName: \"kubernetes.io/projected/e36c8273-fbb4-45dc-bec2-087f65da1f46-kube-api-access-hjk9m\") pod \"payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls\" (UID: \"e36c8273-fbb4-45dc-bec2-087f65da1f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:38:47.849230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.849202 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e36c8273-fbb4-45dc-bec2-087f65da1f46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls\" (UID: \"e36c8273-fbb4-45dc-bec2-087f65da1f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:38:47.872811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.872778 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjk9m\" (UniqueName: \"kubernetes.io/projected/e36c8273-fbb4-45dc-bec2-087f65da1f46-kube-api-access-hjk9m\") pod \"payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls\" (UID: \"e36c8273-fbb4-45dc-bec2-087f65da1f46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:38:47.901071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.901026 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:38:47.950236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.950195 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82svc\" (UniqueName: \"kubernetes.io/projected/bc4521b5-5894-44dc-9419-8cb97d8dcba1-kube-api-access-82svc\") pod \"payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w\" (UID: \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:38:47.950590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.950562 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4521b5-5894-44dc-9419-8cb97d8dcba1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w\" (UID: \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:38:47.951634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.951603 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4521b5-5894-44dc-9419-8cb97d8dcba1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w\" (UID: \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:38:47.965882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.965853 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:38:47.981271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.981237 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82svc\" (UniqueName: \"kubernetes.io/projected/bc4521b5-5894-44dc-9419-8cb97d8dcba1-kube-api-access-82svc\") pod \"payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w\" (UID: \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:38:47.993945 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:47.993646 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:38:48.036764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:48.036731 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:38:48.261252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:48.261218 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls"] Oct 31 21:38:48.261400 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:48.261216 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36c8273_fbb4_45dc_bec2_087f65da1f46.slice/crio-cce932553667c63168531f21f88adf443a390fddfc24ae40463102f00a33ed99 WatchSource:0}: Error finding container cce932553667c63168531f21f88adf443a390fddfc24ae40463102f00a33ed99: Status 404 returned error can't find the container with id cce932553667c63168531f21f88adf443a390fddfc24ae40463102f00a33ed99 Oct 31 21:38:48.377707 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:48.377574 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w"] Oct 31 21:38:48.381271 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:48.381219 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4521b5_5894_44dc_9419_8cb97d8dcba1.slice/crio-b21b933130dd736726645d9cbc27c241b4bbe7c9218bb41215ce107c7bdebb8d WatchSource:0}: Error finding container b21b933130dd736726645d9cbc27c241b4bbe7c9218bb41215ce107c7bdebb8d: Status 404 returned error can't find the container with id b21b933130dd736726645d9cbc27c241b4bbe7c9218bb41215ce107c7bdebb8d Oct 31 21:38:49.139802 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:49.139754 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" event={"ID":"e36c8273-fbb4-45dc-bec2-087f65da1f46","Type":"ContainerStarted","Data":"71060fef9edf3e13de11a0d9f42d7919a26c38c6da59b65a8e5bd454ad1a520c"} Oct 31 21:38:49.140232 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:49.139863 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:38:49.140232 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:49.139885 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" event={"ID":"e36c8273-fbb4-45dc-bec2-087f65da1f46","Type":"ContainerStarted","Data":"cce932553667c63168531f21f88adf443a390fddfc24ae40463102f00a33ed99"} Oct 31 21:38:49.141708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:49.141646 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" event={"ID":"bc4521b5-5894-44dc-9419-8cb97d8dcba1","Type":"ContainerStarted","Data":"5f9250e4b2ed8b4f7fbb3f52262122504c9fe7ca6b5a9bdf92bb7fc6aeba5cff"} Oct 31 21:38:49.141851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:49.141735 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" event={"ID":"bc4521b5-5894-44dc-9419-8cb97d8dcba1","Type":"ContainerStarted","Data":"b21b933130dd736726645d9cbc27c241b4bbe7c9218bb41215ce107c7bdebb8d"} Oct 31 21:38:49.141851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:49.141761 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:38:49.181595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:49.181539 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" podStartSLOduration=2.181520717 podStartE2EDuration="2.181520717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:49.178529358 +0000 UTC m=+1375.871486761" watchObservedRunningTime="2026-10-31 21:38:49.181520717 +0000 UTC m=+1375.874478107" Oct 31 21:38:49.226504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:49.226438 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" podStartSLOduration=2.226422651 podStartE2EDuration="2.226422651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:49.224211789 +0000 UTC m=+1375.917169181" watchObservedRunningTime="2026-10-31 21:38:49.226422651 +0000 UTC m=+1375.919380045" Oct 31 21:38:54.722139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:54.722091 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q"] Oct 31 21:38:54.722636 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:54.722374 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" containerID="cri-o://0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495" gracePeriod=30 Oct 31 21:38:54.733544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:54.733504 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w"] Oct 31 21:38:54.733832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:54.733784 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" containerID="cri-o://d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367" gracePeriod=30 Oct 31 21:38:54.944005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:54.943973 5188 scope.go:122] "RemoveContainer" containerID="1d864ea87c9f1ac8824f2da6b6c070458581693a7cb443c0a09a7bdb3b6e9e30" Oct 31 21:38:54.958257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:54.958192 5188 scope.go:122] "RemoveContainer" containerID="0e5091c44fde7b177b663bdeb9c0a2b86b96fb4e2473b4c44f81c879d28915c9" Oct 31 21:38:54.969032 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:54.969003 5188 scope.go:122] "RemoveContainer" containerID="fa8d231a45817bba9af88a90211933dc7376344b6dbd9c8d17bf945f31fc9a24" Oct 31 21:38:54.982155 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:54.981184 5188 scope.go:122] "RemoveContainer" containerID="bef15e05841a0886c790269932f0693fc6d42a22a2a408dd449216e4e715fae8" Oct 31 21:38:55.000117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:55.000090 5188 scope.go:122] "RemoveContainer" containerID="7bbfa9c87e6527f3268d0f031b42f34eeb86c4ae1d867be483dd7344a458936a" Oct 31 21:38:55.023411 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:55.023384 5188 scope.go:122] "RemoveContainer" containerID="6a8417eea0bf8cb417678eb882badbde196dba9a377f605f03ccb981792d10b4" Oct 31 21:38:55.045671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:55.045228 5188 scope.go:122] "RemoveContainer" containerID="0ae8cfb1b9e80e8f08400f6fccaaa1c3f65920b4e4aed05eac77e9a0589d189d" Oct 31 21:38:57.791228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.791174 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q"] Oct 31 21:38:57.801342 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.799626 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.831722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.826237 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a3dae1-openshift-default-dockercfg-lln64\"" Oct 31 21:38:57.866118 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.866077 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q"] Oct 31 21:38:57.866944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.866910 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a9a747-3941-47bf-8765-fe70dd146648-istio-podinfo\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.867122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.866991 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-envoy\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.867122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.867049 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-istio-token\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.867122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.867083 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-data\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.867307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.867150 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-socket\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.867307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.867202 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkrw\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-kube-api-access-cmkrw\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.867538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.867310 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-credential-socket\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.867538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.867371 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a9a747-3941-47bf-8765-fe70dd146648-istiod-ca-cert\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.867538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.867436 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-certs\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.971350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.971154 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-credential-socket\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.971350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.971215 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a9a747-3941-47bf-8765-fe70dd146648-istiod-ca-cert\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.971350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.971273 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-certs\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.971350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.971356 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a9a747-3941-47bf-8765-fe70dd146648-istio-podinfo\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.971706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.971420 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-envoy\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.971706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.971479 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-istio-token\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.971706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.971506 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-data\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.971706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.971587 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-socket\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.971706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.971634 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkrw\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-kube-api-access-cmkrw\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.977064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.975892 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a9a747-3941-47bf-8765-fe70dd146648-istiod-ca-cert\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.977064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.976120 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-certs\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.977064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.976487 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-credential-socket\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.977064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.976868 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-data\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.977498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.977466 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-socket\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.991671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.980165 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-envoy\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:57.994597 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.994412 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a9a747-3941-47bf-8765-fe70dd146648-istio-podinfo\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:58.002442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:57.997293 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkrw\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-kube-api-access-cmkrw\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:58.050468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:58.048252 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-istio-token\") pod \"e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:58.174232 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:58.174199 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:38:58.532289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:58.532261 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q"] Oct 31 21:38:58.533137 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:38:58.533108 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a9a747_3941_47bf_8765_fe70dd146648.slice/crio-6ae22eba0bc0f6b3f38ce973bbc1d4ea67c597aaa59f9f33c0fbaed250a3a1fe WatchSource:0}: Error finding container 6ae22eba0bc0f6b3f38ce973bbc1d4ea67c597aaa59f9f33c0fbaed250a3a1fe: Status 404 returned error can't find the container with id 6ae22eba0bc0f6b3f38ce973bbc1d4ea67c597aaa59f9f33c0fbaed250a3a1fe Oct 31 21:38:58.536032 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:58.535997 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:38:58.536113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:58.536084 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:38:58.536155 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:58.536116 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:38:59.326094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:59.326054 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" event={"ID":"b0a9a747-3941-47bf-8765-fe70dd146648","Type":"ContainerStarted","Data":"929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6"} Oct 31 21:38:59.326571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:59.326120 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" event={"ID":"b0a9a747-3941-47bf-8765-fe70dd146648","Type":"ContainerStarted","Data":"6ae22eba0bc0f6b3f38ce973bbc1d4ea67c597aaa59f9f33c0fbaed250a3a1fe"} Oct 31 21:38:59.390840 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:38:59.390272 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" podStartSLOduration=2.390248351 podStartE2EDuration="2.390248351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:59.377971991 +0000 UTC m=+1386.070929384" watchObservedRunningTime="2026-10-31 21:38:59.390248351 +0000 UTC m=+1386.083205757" Oct 31 21:39:00.053372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.052607 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:39:00.107587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.107527 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:39:00.111533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.111481 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istiod-ca-cert\") pod \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " Oct 31 21:39:00.111855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.111836 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-token\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-token\") pod \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " Oct 31 21:39:00.112014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.111991 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-certs\") pod \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " Oct 31 21:39:00.112222 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.112200 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-data\") pod \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " Oct 31 21:39:00.112416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.112372 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-envoy\") pod \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " Oct 31 21:39:00.112641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.112615 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-podinfo\") pod \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " Oct 31 21:39:00.113008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.112979 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-socket\") pod \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " Oct 31 21:39:00.113161 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.113137 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-credential-socket\") pod \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " Oct 31 21:39:00.113297 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.113275 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-kube-api-access-zz4lp\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-kube-api-access-zz4lp\") pod \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\" (UID: \"62ad8326-d6e9-4750-ac21-bb1d9c35ca10\") " Oct 31 21:39:00.117948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.113056 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-certs" pod "62ad8326-d6e9-4750-ac21-bb1d9c35ca10" (UID: "62ad8326-d6e9-4750-ac21-bb1d9c35ca10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.117948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.117799 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-credential-socket" pod "62ad8326-d6e9-4750-ac21-bb1d9c35ca10" (UID: "62ad8326-d6e9-4750-ac21-bb1d9c35ca10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.117948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.117873 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-socket" pod "62ad8326-d6e9-4750-ac21-bb1d9c35ca10" (UID: "62ad8326-d6e9-4750-ac21-bb1d9c35ca10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.117948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.117922 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-data" pod "62ad8326-d6e9-4750-ac21-bb1d9c35ca10" (UID: "62ad8326-d6e9-4750-ac21-bb1d9c35ca10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.122980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.122507 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istiod-ca-cert" pod "62ad8326-d6e9-4750-ac21-bb1d9c35ca10" (UID: "62ad8326-d6e9-4750-ac21-bb1d9c35ca10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:39:00.123850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.123807 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-podinfo" pod "62ad8326-d6e9-4750-ac21-bb1d9c35ca10" (UID: "62ad8326-d6e9-4750-ac21-bb1d9c35ca10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:39:00.123850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.123826 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-kube-api-access-zz4lp" pod "62ad8326-d6e9-4750-ac21-bb1d9c35ca10" (UID: "62ad8326-d6e9-4750-ac21-bb1d9c35ca10"). InnerVolumeSpecName "kube-api-access-zz4lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:00.124395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.124293 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-envoy" pod "62ad8326-d6e9-4750-ac21-bb1d9c35ca10" (UID: "62ad8326-d6e9-4750-ac21-bb1d9c35ca10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.126775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.126744 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-token" pod "62ad8326-d6e9-4750-ac21-bb1d9c35ca10" (UID: "62ad8326-d6e9-4750-ac21-bb1d9c35ca10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:00.153779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.153744 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:39:00.153941 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.153842 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:39:00.174788 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.174749 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:39:00.182924 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.182890 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:39:00.218182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.216926 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-certs\") pod \"90363274-a646-4f12-87f9-21439d37625c\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " Oct 31 21:39:00.218182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.217032 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-kube-api-access-dvwjs\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-kube-api-access-dvwjs\") pod \"90363274-a646-4f12-87f9-21439d37625c\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " Oct 31 21:39:00.218182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.217091 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-envoy\") pod \"90363274-a646-4f12-87f9-21439d37625c\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " Oct 31 21:39:00.218182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.217131 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-credential-socket\") pod \"90363274-a646-4f12-87f9-21439d37625c\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " Oct 31 21:39:00.218182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.217184 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-data\") pod \"90363274-a646-4f12-87f9-21439d37625c\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " Oct 31 21:39:00.218182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.217213 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-istio-token\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-istio-token\") pod \"90363274-a646-4f12-87f9-21439d37625c\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " Oct 31 21:39:00.218182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.217277 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90363274-a646-4f12-87f9-21439d37625c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90363274-a646-4f12-87f9-21439d37625c-istio-podinfo\") pod \"90363274-a646-4f12-87f9-21439d37625c\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " Oct 31 21:39:00.218182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.217393 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-socket\") pod \"90363274-a646-4f12-87f9-21439d37625c\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " Oct 31 21:39:00.218182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.217502 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90363274-a646-4f12-87f9-21439d37625c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90363274-a646-4f12-87f9-21439d37625c-istiod-ca-cert\") pod \"90363274-a646-4f12-87f9-21439d37625c\" (UID: \"90363274-a646-4f12-87f9-21439d37625c\") " Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.219831 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-certs" pod "90363274-a646-4f12-87f9-21439d37625c" (UID: "90363274-a646-4f12-87f9-21439d37625c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.219895 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.219922 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.219935 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.219949 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.219963 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz4lp\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-kube-api-access-zz4lp\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.219979 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.219992 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.220004 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.221182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.220015 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62ad8326-d6e9-4750-ac21-bb1d9c35ca10-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.221881 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.221766 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-envoy" pod "90363274-a646-4f12-87f9-21439d37625c" (UID: "90363274-a646-4f12-87f9-21439d37625c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.222408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.222113 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90363274-a646-4f12-87f9-21439d37625c-istiod-ca-cert" pod "90363274-a646-4f12-87f9-21439d37625c" (UID: "90363274-a646-4f12-87f9-21439d37625c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:39:00.222408 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.222191 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90363274-a646-4f12-87f9-21439d37625c-istio-podinfo" pod "90363274-a646-4f12-87f9-21439d37625c" (UID: "90363274-a646-4f12-87f9-21439d37625c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:39:00.222577 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.222468 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-data" pod "90363274-a646-4f12-87f9-21439d37625c" (UID: "90363274-a646-4f12-87f9-21439d37625c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.222720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.222689 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-socket" pod "90363274-a646-4f12-87f9-21439d37625c" (UID: "90363274-a646-4f12-87f9-21439d37625c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.222837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.222761 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-credential-socket" pod "90363274-a646-4f12-87f9-21439d37625c" (UID: "90363274-a646-4f12-87f9-21439d37625c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:00.223648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.223622 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-kube-api-access-dvwjs" pod "90363274-a646-4f12-87f9-21439d37625c" (UID: "90363274-a646-4f12-87f9-21439d37625c"). InnerVolumeSpecName "kube-api-access-dvwjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:00.224780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.224760 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-istio-token" pod "90363274-a646-4f12-87f9-21439d37625c" (UID: "90363274-a646-4f12-87f9-21439d37625c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:00.321882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.321784 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.321882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.321819 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvwjs\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-kube-api-access-dvwjs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.321882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.321829 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.321882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.321838 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.321882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.321846 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.321882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.321856 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90363274-a646-4f12-87f9-21439d37625c-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.321882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.321864 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90363274-a646-4f12-87f9-21439d37625c-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.321882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.321872 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90363274-a646-4f12-87f9-21439d37625c-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.321882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.321880 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90363274-a646-4f12-87f9-21439d37625c-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:00.338112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.338076 5188 generic.go:356] "Generic (PLEG): container finished" podID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerID="0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495" exitCode=0 Oct 31 21:39:00.338682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.338124 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" Oct 31 21:39:00.338682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.338150 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" event={"ID":"62ad8326-d6e9-4750-ac21-bb1d9c35ca10","Type":"ContainerDied","Data":"0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495"} Oct 31 21:39:00.338682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.338215 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q" event={"ID":"62ad8326-d6e9-4750-ac21-bb1d9c35ca10","Type":"ContainerDied","Data":"3a0f300312732d7f225402140eac10eab45e1c3835ddbfb0e0a09452b35eabb3"} Oct 31 21:39:00.338682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.338256 5188 scope.go:122] "RemoveContainer" containerID="0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495" Oct 31 21:39:00.340238 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.340212 5188 generic.go:356] "Generic (PLEG): container finished" podID="90363274-a646-4f12-87f9-21439d37625c" containerID="d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367" exitCode=0 Oct 31 21:39:00.341431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.341410 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" Oct 31 21:39:00.345449 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.345419 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" event={"ID":"90363274-a646-4f12-87f9-21439d37625c","Type":"ContainerDied","Data":"d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367"} Oct 31 21:39:00.345566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.345488 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:39:00.345566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.345519 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w" event={"ID":"90363274-a646-4f12-87f9-21439d37625c","Type":"ContainerDied","Data":"0eb1f0ebff9b71550aadb3911881d0adeb74369087a4416d01eebe6746025396"} Oct 31 21:39:00.346449 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.346428 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:39:00.351968 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.351949 5188 scope.go:122] "RemoveContainer" containerID="0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495" Oct 31 21:39:00.352266 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:39:00.352249 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495\": container with ID starting with 0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495 not found: ID does not exist" containerID="0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495" Oct 31 21:39:00.352365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.352272 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495"} err="failed to get container status \"0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495\": rpc error: code = NotFound desc = could not find container \"0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495\": container with ID starting with 0032d1287823d401f251277dc707f26564f39023b9a12055b1323d14b132b495 not found: ID does not exist" Oct 31 21:39:00.352365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.352288 5188 scope.go:122] "RemoveContainer" containerID="d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367" Oct 31 21:39:00.362994 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.362969 5188 scope.go:122] "RemoveContainer" containerID="d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367" Oct 31 21:39:00.363358 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:39:00.363333 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367\": container with ID starting with d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367 not found: ID does not exist" containerID="d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367" Oct 31 21:39:00.363428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.363371 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367"} err="failed to get container status \"d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367\": rpc error: code = NotFound desc = could not find container \"d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367\": container with ID starting with d1b0b0f557395eae1d7bba9e26482c890a89733dbb6e067f9ac5181faaa05367 not found: ID does not exist" Oct 31 21:39:00.444452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.444387 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q"] Oct 31 21:39:00.448463 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.448430 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-5h64q"] Oct 31 21:39:00.464946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.464915 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w"] Oct 31 21:39:00.469192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:00.469161 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-lvv4w"] Oct 31 21:39:01.221535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.221495 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx"] Oct 31 21:39:01.222822 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.222792 5188 cpu_manager.go:414] "containerMap: removing container" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" Oct 31 21:39:01.222992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.222978 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" Oct 31 21:39:01.223104 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.223092 5188 cpu_manager.go:414] "containerMap: removing container" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" Oct 31 21:39:01.223187 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.223176 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" Oct 31 21:39:01.223490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.223472 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" containerName="istio-proxy" Oct 31 21:39:01.224093 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.224073 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="90363274-a646-4f12-87f9-21439d37625c" containerName="istio-proxy" Oct 31 21:39:01.230973 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.230945 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:01.248372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.245936 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx"] Oct 31 21:39:01.248372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.247577 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c40fc3\"" Oct 31 21:39:01.248372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.247711 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c40fc3-dockercfg-q9kr6\"" Oct 31 21:39:01.344533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.344501 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx\" (UID: \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:01.345011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.344759 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wjmd\" (UniqueName: \"kubernetes.io/projected/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-kube-api-access-4wjmd\") pod \"payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx\" (UID: \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:01.346072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.346049 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v"] Oct 31 21:39:01.350547 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.350527 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:01.371044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.371013 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v"] Oct 31 21:39:01.446016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.445971 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr4rv\" (UniqueName: \"kubernetes.io/projected/892582bc-6881-4e05-be5d-1fc72dc82847-kube-api-access-zr4rv\") pod \"payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v\" (UID: \"892582bc-6881-4e05-be5d-1fc72dc82847\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:01.446202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.446116 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wjmd\" (UniqueName: \"kubernetes.io/projected/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-kube-api-access-4wjmd\") pod \"payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx\" (UID: \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:01.446202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.446191 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892582bc-6881-4e05-be5d-1fc72dc82847-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v\" (UID: \"892582bc-6881-4e05-be5d-1fc72dc82847\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:01.446311 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.446277 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx\" (UID: \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:01.447139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.447116 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx\" (UID: \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:01.468312 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.468278 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wjmd\" (UniqueName: \"kubernetes.io/projected/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-kube-api-access-4wjmd\") pod \"payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx\" (UID: \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:01.475723 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.475688 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ad8326-d6e9-4750-ac21-bb1d9c35ca10" path="/var/lib/kubelet/pods/62ad8326-d6e9-4750-ac21-bb1d9c35ca10/volumes" Oct 31 21:39:01.476301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.476268 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90363274-a646-4f12-87f9-21439d37625c" path="/var/lib/kubelet/pods/90363274-a646-4f12-87f9-21439d37625c/volumes" Oct 31 21:39:01.548090 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.547997 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr4rv\" (UniqueName: \"kubernetes.io/projected/892582bc-6881-4e05-be5d-1fc72dc82847-kube-api-access-zr4rv\") pod \"payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v\" (UID: \"892582bc-6881-4e05-be5d-1fc72dc82847\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:01.548276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.548141 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892582bc-6881-4e05-be5d-1fc72dc82847-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v\" (UID: \"892582bc-6881-4e05-be5d-1fc72dc82847\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:01.548931 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.548913 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892582bc-6881-4e05-be5d-1fc72dc82847-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v\" (UID: \"892582bc-6881-4e05-be5d-1fc72dc82847\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:01.570483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.570451 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr4rv\" (UniqueName: \"kubernetes.io/projected/892582bc-6881-4e05-be5d-1fc72dc82847-kube-api-access-zr4rv\") pod \"payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v\" (UID: \"892582bc-6881-4e05-be5d-1fc72dc82847\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:01.590645 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.590605 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:01.667902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.667560 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:01.908915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:01.908869 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx"] Oct 31 21:39:02.057741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:02.057706 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v"] Oct 31 21:39:02.058995 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:39:02.058963 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892582bc_6881_4e05_be5d_1fc72dc82847.slice/crio-e2a782287da7c6653d4cd8093467194b64aa2560067db65cb7aa20ec850fa643 WatchSource:0}: Error finding container e2a782287da7c6653d4cd8093467194b64aa2560067db65cb7aa20ec850fa643: Status 404 returned error can't find the container with id e2a782287da7c6653d4cd8093467194b64aa2560067db65cb7aa20ec850fa643 Oct 31 21:39:02.364774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:02.364725 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" event={"ID":"892582bc-6881-4e05-be5d-1fc72dc82847","Type":"ContainerStarted","Data":"70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f"} Oct 31 21:39:02.365226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:02.364805 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:02.365226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:02.364840 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" event={"ID":"892582bc-6881-4e05-be5d-1fc72dc82847","Type":"ContainerStarted","Data":"e2a782287da7c6653d4cd8093467194b64aa2560067db65cb7aa20ec850fa643"} Oct 31 21:39:02.366630 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:02.366604 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" event={"ID":"2d2107fc-814c-4a1f-adfa-e2023e8c52bf","Type":"ContainerStarted","Data":"10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d"} Oct 31 21:39:02.366773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:02.366651 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" event={"ID":"2d2107fc-814c-4a1f-adfa-e2023e8c52bf","Type":"ContainerStarted","Data":"4a77416df745f9a520bd9937cef258c0dab03c06ff014c5f15a9568e908f71e1"} Oct 31 21:39:02.366773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:02.366692 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:02.461868 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:02.461800 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" podStartSLOduration=1.461785911 podStartE2EDuration="1.461785911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:02.408168437 +0000 UTC m=+1389.101125831" watchObservedRunningTime="2026-10-31 21:39:02.461785911 +0000 UTC m=+1389.154743306" Oct 31 21:39:02.462648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:02.462598 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" podStartSLOduration=1.46258468 podStartE2EDuration="1.46258468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:02.461579293 +0000 UTC m=+1389.154536686" watchObservedRunningTime="2026-10-31 21:39:02.46258468 +0000 UTC m=+1389.155542075" Oct 31 21:39:02.542615 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:39:02.542572 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:39:04.395763 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.395507 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h"] Oct 31 21:39:04.401618 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.401588 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:04.422475 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.422445 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-89d4a2-dockercfg-48mv8\"" Oct 31 21:39:04.428850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.428820 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-89d4a2\"" Oct 31 21:39:04.434655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.434563 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h"] Oct 31 21:39:04.538265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.538232 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdhf\" (UniqueName: \"kubernetes.io/projected/084b5c8c-54a8-4319-865d-3d1f43811abd-kube-api-access-6pdhf\") pod \"payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h\" (UID: \"084b5c8c-54a8-4319-865d-3d1f43811abd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:04.538446 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.538358 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084b5c8c-54a8-4319-865d-3d1f43811abd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h\" (UID: \"084b5c8c-54a8-4319-865d-3d1f43811abd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:04.574965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.574928 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk"] Oct 31 21:39:04.585396 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.585367 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:04.666402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.653760 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdhf\" (UniqueName: \"kubernetes.io/projected/084b5c8c-54a8-4319-865d-3d1f43811abd-kube-api-access-6pdhf\") pod \"payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h\" (UID: \"084b5c8c-54a8-4319-865d-3d1f43811abd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:04.690412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.688784 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084b5c8c-54a8-4319-865d-3d1f43811abd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h\" (UID: \"084b5c8c-54a8-4319-865d-3d1f43811abd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:04.690412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.686739 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk"] Oct 31 21:39:04.698757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.697815 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084b5c8c-54a8-4319-865d-3d1f43811abd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h\" (UID: \"084b5c8c-54a8-4319-865d-3d1f43811abd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:04.703646 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.703580 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdhf\" (UniqueName: \"kubernetes.io/projected/084b5c8c-54a8-4319-865d-3d1f43811abd-kube-api-access-6pdhf\") pod \"payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h\" (UID: \"084b5c8c-54a8-4319-865d-3d1f43811abd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:04.733066 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.732752 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:04.792115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.792074 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7zd\" (UniqueName: \"kubernetes.io/projected/fa027455-c262-4144-9928-d9c0c43fd335-kube-api-access-9q7zd\") pod \"payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk\" (UID: \"fa027455-c262-4144-9928-d9c0c43fd335\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:04.792365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.792156 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa027455-c262-4144-9928-d9c0c43fd335-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk\" (UID: \"fa027455-c262-4144-9928-d9c0c43fd335\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:04.824188 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.824142 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs"] Oct 31 21:39:04.833681 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.833314 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:04.841409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.841373 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs"] Oct 31 21:39:04.847124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.846845 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a3dae1-dockercfg-9w2fb\"" Oct 31 21:39:04.847124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.846954 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a3dae1\"" Oct 31 21:39:04.898345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.894161 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa027455-c262-4144-9928-d9c0c43fd335-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk\" (UID: \"fa027455-c262-4144-9928-d9c0c43fd335\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:04.898345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.894409 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs\" (UID: \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:04.898345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.894501 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7zd\" (UniqueName: \"kubernetes.io/projected/fa027455-c262-4144-9928-d9c0c43fd335-kube-api-access-9q7zd\") pod \"payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk\" (UID: \"fa027455-c262-4144-9928-d9c0c43fd335\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:04.898345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.894533 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t56q\" (UniqueName: \"kubernetes.io/projected/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-kube-api-access-6t56q\") pod \"payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs\" (UID: \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:04.898345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.895464 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa027455-c262-4144-9928-d9c0c43fd335-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk\" (UID: \"fa027455-c262-4144-9928-d9c0c43fd335\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:04.937547 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:04.937417 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7zd\" (UniqueName: \"kubernetes.io/projected/fa027455-c262-4144-9928-d9c0c43fd335-kube-api-access-9q7zd\") pod \"payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk\" (UID: \"fa027455-c262-4144-9928-d9c0c43fd335\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:05.082721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.079393 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj"] Oct 31 21:39:05.084251 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.084213 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs\" (UID: \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:05.084435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.084354 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t56q\" (UniqueName: \"kubernetes.io/projected/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-kube-api-access-6t56q\") pod \"payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs\" (UID: \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:05.086262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.086239 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:05.104029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.103994 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs\" (UID: \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:05.104516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.104493 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj"] Oct 31 21:39:05.121877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.121831 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t56q\" (UniqueName: \"kubernetes.io/projected/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-kube-api-access-6t56q\") pod \"payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs\" (UID: \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:05.159153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.158913 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:05.163677 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.163652 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h"] Oct 31 21:39:05.163933 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:39:05.163901 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084b5c8c_54a8_4319_865d_3d1f43811abd.slice/crio-7955ee1c27657922b70d1906890dae6bc0da49e2f75bf010f146e7c3ead0e048 WatchSource:0}: Error finding container 7955ee1c27657922b70d1906890dae6bc0da49e2f75bf010f146e7c3ead0e048: Status 404 returned error can't find the container with id 7955ee1c27657922b70d1906890dae6bc0da49e2f75bf010f146e7c3ead0e048 Oct 31 21:39:05.186881 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.185289 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqznc\" (UniqueName: \"kubernetes.io/projected/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-kube-api-access-qqznc\") pod \"payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj\" (UID: \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:05.186881 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.185507 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj\" (UID: \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:05.224638 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.224602 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:05.296685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.295389 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqznc\" (UniqueName: \"kubernetes.io/projected/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-kube-api-access-qqznc\") pod \"payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj\" (UID: \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:05.296685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.295596 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj\" (UID: \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:05.296685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.296489 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj\" (UID: \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:05.356479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.356446 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqznc\" (UniqueName: \"kubernetes.io/projected/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-kube-api-access-qqznc\") pod \"payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj\" (UID: \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:05.410360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.409663 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" event={"ID":"084b5c8c-54a8-4319-865d-3d1f43811abd","Type":"ContainerStarted","Data":"df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2"} Oct 31 21:39:05.410360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.409736 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" event={"ID":"084b5c8c-54a8-4319-865d-3d1f43811abd","Type":"ContainerStarted","Data":"7955ee1c27657922b70d1906890dae6bc0da49e2f75bf010f146e7c3ead0e048"} Oct 31 21:39:05.410360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.409975 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:05.439438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.439402 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:05.458574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.458341 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" podStartSLOduration=1.458306627 podStartE2EDuration="1.458306627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:05.456854908 +0000 UTC m=+1392.149812302" watchObservedRunningTime="2026-10-31 21:39:05.458306627 +0000 UTC m=+1392.151264023" Oct 31 21:39:05.522607 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.522154 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs"] Oct 31 21:39:05.524649 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:39:05.524612 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f49e1e_4677_49e1_9bcf_f5fe9ceea30a.slice/crio-39ddf6f43bd6d755311e39cead88a7d2d9e23f4ea2c6f6bfb05a45bf06edb5e7 WatchSource:0}: Error finding container 39ddf6f43bd6d755311e39cead88a7d2d9e23f4ea2c6f6bfb05a45bf06edb5e7: Status 404 returned error can't find the container with id 39ddf6f43bd6d755311e39cead88a7d2d9e23f4ea2c6f6bfb05a45bf06edb5e7 Oct 31 21:39:05.594116 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:39:05.594059 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa027455_c262_4144_9928_d9c0c43fd335.slice/crio-49b001e0247383253c88eedd0b9d31ef69f641a090eff0ad90a8387a47eb1ac2 WatchSource:0}: Error finding container 49b001e0247383253c88eedd0b9d31ef69f641a090eff0ad90a8387a47eb1ac2: Status 404 returned error can't find the container with id 49b001e0247383253c88eedd0b9d31ef69f641a090eff0ad90a8387a47eb1ac2 Oct 31 21:39:05.606577 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.606540 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk"] Oct 31 21:39:05.775977 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:05.775700 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj"] Oct 31 21:39:06.422430 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.422383 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" event={"ID":"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a","Type":"ContainerStarted","Data":"0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8"} Oct 31 21:39:06.423105 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.422448 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" event={"ID":"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a","Type":"ContainerStarted","Data":"39ddf6f43bd6d755311e39cead88a7d2d9e23f4ea2c6f6bfb05a45bf06edb5e7"} Oct 31 21:39:06.423105 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.422760 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:06.425311 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.425282 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" event={"ID":"fa027455-c262-4144-9928-d9c0c43fd335","Type":"ContainerStarted","Data":"6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039"} Oct 31 21:39:06.425454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.425346 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" event={"ID":"fa027455-c262-4144-9928-d9c0c43fd335","Type":"ContainerStarted","Data":"49b001e0247383253c88eedd0b9d31ef69f641a090eff0ad90a8387a47eb1ac2"} Oct 31 21:39:06.425454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.425429 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:06.427450 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.427425 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" event={"ID":"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d","Type":"ContainerStarted","Data":"ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5"} Oct 31 21:39:06.427583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.427471 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" event={"ID":"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d","Type":"ContainerStarted","Data":"a95f23366023739921008c7984f8960f889e39ebe227cd55e2b54e3387e6c965"} Oct 31 21:39:06.427583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.427571 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:06.475620 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.475551 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" podStartSLOduration=2.475525848 podStartE2EDuration="2.475525848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:06.469442928 +0000 UTC m=+1393.162400325" watchObservedRunningTime="2026-10-31 21:39:06.475525848 +0000 UTC m=+1393.168483245" Oct 31 21:39:06.514637 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.514569 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" podStartSLOduration=2.514550561 podStartE2EDuration="2.514550561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:06.512198108 +0000 UTC m=+1393.205155501" watchObservedRunningTime="2026-10-31 21:39:06.514550561 +0000 UTC m=+1393.207507946" Oct 31 21:39:06.568998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:06.568942 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" podStartSLOduration=2.568926723 podStartE2EDuration="2.568926723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:06.566512599 +0000 UTC m=+1393.259469993" watchObservedRunningTime="2026-10-31 21:39:06.568926723 +0000 UTC m=+1393.261884116" Oct 31 21:39:09.033570 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:09.033535 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh"] Oct 31 21:39:09.042816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:09.042783 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f51fc3-z28vh"] Oct 31 21:39:09.475565 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:09.475520 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1fc46c-dbbc-43c6-9fc4-920caf037cf1" path="/var/lib/kubelet/pods/ed1fc46c-dbbc-43c6-9fc4-920caf037cf1/volumes" Oct 31 21:39:12.821992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:12.821942 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957"] Oct 31 21:39:12.826890 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:12.826858 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:12.840039 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:12.840002 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-752623-serving-cert\"" Oct 31 21:39:12.856589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:12.856545 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957"] Oct 31 21:39:12.911845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:12.911804 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-metrics-certs\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:12.912082 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:12.912054 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917b637a-293e-4d77-8300-2369298b744f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:12.912165 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:12.912124 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwgnw\" (UniqueName: \"kubernetes.io/projected/917b637a-293e-4d77-8300-2369298b744f-kube-api-access-nwgnw\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:12.912228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:12.912163 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.013518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.013454 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917b637a-293e-4d77-8300-2369298b744f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.013749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.013540 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwgnw\" (UniqueName: \"kubernetes.io/projected/917b637a-293e-4d77-8300-2369298b744f-kube-api-access-nwgnw\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.013749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.013579 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.013749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.013717 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-metrics-certs\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.014257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.014229 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917b637a-293e-4d77-8300-2369298b744f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.016508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.016477 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.016664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.016600 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-metrics-certs\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.041993 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.041953 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwgnw\" (UniqueName: \"kubernetes.io/projected/917b637a-293e-4d77-8300-2369298b744f-kube-api-access-nwgnw\") pod \"maas-api-e2e-ipp-b-752623-6578479477-7c957\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.146309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.146263 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:13.380537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.380500 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:39:13.380715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.380585 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:39:13.475573 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.475528 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:39:13.497191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.497152 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957"] Oct 31 21:39:13.508442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:13.508393 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" event={"ID":"917b637a-293e-4d77-8300-2369298b744f","Type":"ContainerStarted","Data":"d4e2017e686bfb14133b7876cb8d5e0aa53e223fe90c5b7faf103a402dfd303a"} Oct 31 21:39:14.230384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:14.230303 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2"] Oct 31 21:39:14.281452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:14.281407 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7"] Oct 31 21:39:14.522578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:14.522465 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" event={"ID":"917b637a-293e-4d77-8300-2369298b744f","Type":"ContainerStarted","Data":"8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4"} Oct 31 21:39:14.522578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:14.522555 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:14.562937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:14.562865 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" podStartSLOduration=2.216011596 podStartE2EDuration="2.562842252s" totalImagesPullingTime="346.830656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:14.560210788 +0000 UTC m=+1401.253168181" watchObservedRunningTime="2026-10-31 21:39:14.562842252 +0000 UTC m=+1401.255799645" Oct 31 21:39:16.430672 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:16.430642 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:39:17.439935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:17.439899 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:39:17.441676 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:17.441638 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:39:17.441843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:17.441714 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:39:19.231597 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.231527 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" podUID="40f9027e-c1b9-46b6-bc5a-fb827fc67c6f" containerName="payload-processing" containerID="cri-o://6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be" gracePeriod=25 Oct 31 21:39:19.282677 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.282612 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" podUID="ee63ab77-96a8-4c13-828a-49554f3ef33a" containerName="payload-pre-processing" containerID="cri-o://17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47" gracePeriod=25 Oct 31 21:39:19.637817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.637786 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:39:19.683118 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.682476 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:39:19.689502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.687934 5188 generic.go:356] "Generic (PLEG): container finished" podID="ee63ab77-96a8-4c13-828a-49554f3ef33a" containerID="17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47" exitCode=0 Oct 31 21:39:19.689502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.688143 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" event={"ID":"ee63ab77-96a8-4c13-828a-49554f3ef33a","Type":"ContainerDied","Data":"17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47"} Oct 31 21:39:19.689502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.688196 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" event={"ID":"ee63ab77-96a8-4c13-828a-49554f3ef33a","Type":"ContainerDied","Data":"23e94c03d0be6de767c4c3550b241b71b88e9270195f488fda1fd2ee1b0e0dad"} Oct 31 21:39:19.689502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.688227 5188 scope.go:122] "RemoveContainer" containerID="17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47" Oct 31 21:39:19.689502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.688428 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7" Oct 31 21:39:19.692602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.692568 5188 generic.go:356] "Generic (PLEG): container finished" podID="40f9027e-c1b9-46b6-bc5a-fb827fc67c6f" containerID="6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be" exitCode=0 Oct 31 21:39:19.692870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.692812 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" Oct 31 21:39:19.693146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.692831 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" event={"ID":"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f","Type":"ContainerDied","Data":"6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be"} Oct 31 21:39:19.693292 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.693276 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2" event={"ID":"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f","Type":"ContainerDied","Data":"dbebbc27c7caf9f69d16530172bedb9c71926812550e6a471ad482a4b5e04c09"} Oct 31 21:39:19.708893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.708852 5188 scope.go:122] "RemoveContainer" containerID="17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47" Oct 31 21:39:19.709523 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:39:19.709309 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47\": container with ID starting with 17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47 not found: ID does not exist" containerID="17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47" Oct 31 21:39:19.709523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.709384 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47"} err="failed to get container status \"17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47\": rpc error: code = NotFound desc = could not find container \"17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47\": container with ID starting with 17c27f24f018faf15069bee73f74d2c53e63fa3ef864db738f6a8b57b1e1cd47 not found: ID does not exist" Oct 31 21:39:19.709523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.709410 5188 scope.go:122] "RemoveContainer" containerID="6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be" Oct 31 21:39:19.725692 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.725667 5188 scope.go:122] "RemoveContainer" containerID="6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be" Oct 31 21:39:19.726105 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:39:19.726076 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be\": container with ID starting with 6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be not found: ID does not exist" containerID="6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be" Oct 31 21:39:19.726215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.726141 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be"} err="failed to get container status \"6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be\": rpc error: code = NotFound desc = could not find container \"6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be\": container with ID starting with 6b5163f2d4a4e023bf57466dbeee91dce6d10bbf176cd8f0f14a8d6d1d2562be not found: ID does not exist" Oct 31 21:39:19.762619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.762572 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee63ab77-96a8-4c13-828a-49554f3ef33a-kube-api-access-prsjh\" (UniqueName: \"kubernetes.io/projected/ee63ab77-96a8-4c13-828a-49554f3ef33a-kube-api-access-prsjh\") pod \"ee63ab77-96a8-4c13-828a-49554f3ef33a\" (UID: \"ee63ab77-96a8-4c13-828a-49554f3ef33a\") " Oct 31 21:39:19.762825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.762705 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee63ab77-96a8-4c13-828a-49554f3ef33a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee63ab77-96a8-4c13-828a-49554f3ef33a-plugins-config-volume\") pod \"ee63ab77-96a8-4c13-828a-49554f3ef33a\" (UID: \"ee63ab77-96a8-4c13-828a-49554f3ef33a\") " Oct 31 21:39:19.762825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.762765 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-plugins-config-volume\") pod \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\" (UID: \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\") " Oct 31 21:39:19.762982 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.762955 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-kube-api-access-snxqt\" (UniqueName: \"kubernetes.io/projected/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-kube-api-access-snxqt\") pod \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\" (UID: \"40f9027e-c1b9-46b6-bc5a-fb827fc67c6f\") " Oct 31 21:39:19.763253 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.763190 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee63ab77-96a8-4c13-828a-49554f3ef33a-plugins-config-volume" pod "ee63ab77-96a8-4c13-828a-49554f3ef33a" (UID: "ee63ab77-96a8-4c13-828a-49554f3ef33a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:39:19.763378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.763196 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-plugins-config-volume" pod "40f9027e-c1b9-46b6-bc5a-fb827fc67c6f" (UID: "40f9027e-c1b9-46b6-bc5a-fb827fc67c6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:39:19.763595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.763576 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee63ab77-96a8-4c13-828a-49554f3ef33a-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:19.763595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.763595 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:19.764998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.764972 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee63ab77-96a8-4c13-828a-49554f3ef33a-kube-api-access-prsjh" pod "ee63ab77-96a8-4c13-828a-49554f3ef33a" (UID: "ee63ab77-96a8-4c13-828a-49554f3ef33a"). InnerVolumeSpecName "kube-api-access-prsjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:19.765122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.765099 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-kube-api-access-snxqt" pod "40f9027e-c1b9-46b6-bc5a-fb827fc67c6f" (UID: "40f9027e-c1b9-46b6-bc5a-fb827fc67c6f"). InnerVolumeSpecName "kube-api-access-snxqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:19.864906 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.864869 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prsjh\" (UniqueName: \"kubernetes.io/projected/ee63ab77-96a8-4c13-828a-49554f3ef33a-kube-api-access-prsjh\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:19.864906 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:19.864901 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snxqt\" (UniqueName: \"kubernetes.io/projected/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f-kube-api-access-snxqt\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:20.033964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:20.033924 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7"] Oct 31 21:39:20.038637 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:20.038602 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-66664f4bf5-k9bj7"] Oct 31 21:39:20.078407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:20.078366 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2"] Oct 31 21:39:20.086907 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:20.086147 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-6c8845777f-jdvq2"] Oct 31 21:39:20.536936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:20.536849 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:39:21.475879 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:21.475829 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f9027e-c1b9-46b6-bc5a-fb827fc67c6f" path="/var/lib/kubelet/pods/40f9027e-c1b9-46b6-bc5a-fb827fc67c6f/volumes" Oct 31 21:39:21.476443 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:21.476420 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee63ab77-96a8-4c13-828a-49554f3ef33a" path="/var/lib/kubelet/pods/ee63ab77-96a8-4c13-828a-49554f3ef33a/volumes" Oct 31 21:39:30.805524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.805480 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb"] Oct 31 21:39:30.808064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.806383 5188 cpu_manager.go:414] "containerMap: removing container" podUID="40f9027e-c1b9-46b6-bc5a-fb827fc67c6f" containerName="payload-processing" Oct 31 21:39:30.808064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.806407 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f9027e-c1b9-46b6-bc5a-fb827fc67c6f" containerName="payload-processing" Oct 31 21:39:30.808064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.806439 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ee63ab77-96a8-4c13-828a-49554f3ef33a" containerName="payload-pre-processing" Oct 31 21:39:30.808064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.806445 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee63ab77-96a8-4c13-828a-49554f3ef33a" containerName="payload-pre-processing" Oct 31 21:39:30.808064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.806575 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee63ab77-96a8-4c13-828a-49554f3ef33a" containerName="payload-pre-processing" Oct 31 21:39:30.808064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.806591 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f9027e-c1b9-46b6-bc5a-fb827fc67c6f" containerName="payload-processing" Oct 31 21:39:30.810807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.810785 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:30.835724 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.835688 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a3dae1-serving-cert\"" Oct 31 21:39:30.840202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.840168 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb"] Oct 31 21:39:30.884086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.884030 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-metrics-certs\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:30.884371 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.884338 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-maas-api-tls\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:30.884475 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.884450 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch88v\" (UniqueName: \"kubernetes.io/projected/c57e1750-8e39-4c03-aab6-c58dcb835b0a-kube-api-access-ch88v\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:30.884563 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.884522 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57e1750-8e39-4c03-aab6-c58dcb835b0a-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:30.986668 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.986614 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-maas-api-tls\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:30.986888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.986747 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch88v\" (UniqueName: \"kubernetes.io/projected/c57e1750-8e39-4c03-aab6-c58dcb835b0a-kube-api-access-ch88v\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:30.986888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.986783 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57e1750-8e39-4c03-aab6-c58dcb835b0a-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:30.987010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.986983 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-metrics-certs\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:30.989466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:30.989161 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57e1750-8e39-4c03-aab6-c58dcb835b0a-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:31.036551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:31.035958 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-maas-api-tls\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:31.042589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:31.042539 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch88v\" (UniqueName: \"kubernetes.io/projected/c57e1750-8e39-4c03-aab6-c58dcb835b0a-kube-api-access-ch88v\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:31.043045 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:31.043004 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-metrics-certs\") pod \"maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:31.130669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:31.130633 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:31.444566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:31.444534 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb"] Oct 31 21:39:31.444975 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:39:31.444942 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57e1750_8e39_4c03_aab6_c58dcb835b0a.slice/crio-86ccc13214ca640ff854f5dcb51cd4cdfaa088b7903a0fe90811e6d9a289c4ae WatchSource:0}: Error finding container 86ccc13214ca640ff854f5dcb51cd4cdfaa088b7903a0fe90811e6d9a289c4ae: Status 404 returned error can't find the container with id 86ccc13214ca640ff854f5dcb51cd4cdfaa088b7903a0fe90811e6d9a289c4ae Oct 31 21:39:31.810164 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:31.810117 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" event={"ID":"c57e1750-8e39-4c03-aab6-c58dcb835b0a","Type":"ContainerStarted","Data":"86ccc13214ca640ff854f5dcb51cd4cdfaa088b7903a0fe90811e6d9a289c4ae"} Oct 31 21:39:32.541101 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:39:32.541057 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:39:32.831099 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:32.830982 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" event={"ID":"c57e1750-8e39-4c03-aab6-c58dcb835b0a","Type":"ContainerStarted","Data":"ff1ecdd7c8c1378b05d10172abfdb0a27e49664c3c5e128243cfc5db442c0d0a"} Oct 31 21:39:32.831650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:32.831351 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:32.870651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:32.870567 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" podStartSLOduration=2.573511654 podStartE2EDuration="2.870544458s" totalImagesPullingTime="297.032804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:32.868649937 +0000 UTC m=+1419.561607331" watchObservedRunningTime="2026-10-31 21:39:32.870544458 +0000 UTC m=+1419.563501846" Oct 31 21:39:38.844843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:38.844812 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:39:40.426345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.421373 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb"] Oct 31 21:39:40.492348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.479772 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:40.528355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.527502 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d75c94\"" Oct 31 21:39:40.552752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.552706 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb"] Oct 31 21:39:40.553452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.553426 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d75c94-dockercfg-zbtnt\"" Oct 31 21:39:40.628706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.628665 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb\" (UID: \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:40.630534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.630493 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsbp\" (UniqueName: \"kubernetes.io/projected/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-kube-api-access-jgsbp\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb\" (UID: \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:40.636289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.636260 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8"] Oct 31 21:39:40.645767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.645737 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8"] Oct 31 21:39:40.645912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.645843 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:40.731608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.731512 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp245\" (UniqueName: \"kubernetes.io/projected/7f20f387-c81d-41eb-804b-6b992484f56b-kube-api-access-pp245\") pod \"payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8\" (UID: \"7f20f387-c81d-41eb-804b-6b992484f56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:40.731608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.731553 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f20f387-c81d-41eb-804b-6b992484f56b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8\" (UID: \"7f20f387-c81d-41eb-804b-6b992484f56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:40.731814 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.731674 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb\" (UID: \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:40.731814 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.731763 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsbp\" (UniqueName: \"kubernetes.io/projected/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-kube-api-access-jgsbp\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb\" (UID: \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:40.732680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.732660 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb\" (UID: \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:40.754056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.754018 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsbp\" (UniqueName: \"kubernetes.io/projected/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-kube-api-access-jgsbp\") pod \"payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb\" (UID: \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:40.832925 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.832886 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp245\" (UniqueName: \"kubernetes.io/projected/7f20f387-c81d-41eb-804b-6b992484f56b-kube-api-access-pp245\") pod \"payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8\" (UID: \"7f20f387-c81d-41eb-804b-6b992484f56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:40.832925 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.832929 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f20f387-c81d-41eb-804b-6b992484f56b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8\" (UID: \"7f20f387-c81d-41eb-804b-6b992484f56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:40.833762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.833737 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f20f387-c81d-41eb-804b-6b992484f56b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8\" (UID: \"7f20f387-c81d-41eb-804b-6b992484f56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:40.855202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.855165 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp245\" (UniqueName: \"kubernetes.io/projected/7f20f387-c81d-41eb-804b-6b992484f56b-kube-api-access-pp245\") pod \"payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8\" (UID: \"7f20f387-c81d-41eb-804b-6b992484f56b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:40.943596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.943563 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:40.965442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:40.965401 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:41.310959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:41.310810 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb"] Oct 31 21:39:41.419686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:41.419623 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8"] Oct 31 21:39:41.422679 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:39:41.422621 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f20f387_c81d_41eb_804b_6b992484f56b.slice/crio-d7f90fc1749fa53e53251b73e0c53c3726f4bb83de174514541eb480b4c420c6 WatchSource:0}: Error finding container d7f90fc1749fa53e53251b73e0c53c3726f4bb83de174514541eb480b4c420c6: Status 404 returned error can't find the container with id d7f90fc1749fa53e53251b73e0c53c3726f4bb83de174514541eb480b4c420c6 Oct 31 21:39:41.952187 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:41.952152 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" event={"ID":"82aa5bfd-7b7d-435f-a76f-36c14c0f0251","Type":"ContainerStarted","Data":"dd2041ca1cb1f5dd1233e3955931c5cdab6222ecf9ccf699738ca49f8ce1da6a"} Oct 31 21:39:41.952709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:41.952207 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" event={"ID":"82aa5bfd-7b7d-435f-a76f-36c14c0f0251","Type":"ContainerStarted","Data":"04eff68c149a4e7eb6612cf867444c249b356b75bbb22f728dfd4aab60dad88b"} Oct 31 21:39:41.952891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:41.952718 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:41.954058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:41.954035 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" event={"ID":"7f20f387-c81d-41eb-804b-6b992484f56b","Type":"ContainerStarted","Data":"7099ce4a6fe453355fe0f96027c20fbf6cbf2e703d2735723305abccac9634b5"} Oct 31 21:39:41.954203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:41.954069 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" event={"ID":"7f20f387-c81d-41eb-804b-6b992484f56b","Type":"ContainerStarted","Data":"d7f90fc1749fa53e53251b73e0c53c3726f4bb83de174514541eb480b4c420c6"} Oct 31 21:39:41.954360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:41.954349 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:42.006349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:42.005652 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" podStartSLOduration=2.005627319 podStartE2EDuration="2.005627319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:41.99417052 +0000 UTC m=+1428.687127914" watchObservedRunningTime="2026-10-31 21:39:42.005627319 +0000 UTC m=+1428.698584714" Oct 31 21:39:42.041220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:42.041149 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" podStartSLOduration=2.04113066 podStartE2EDuration="2.04113066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:42.035045828 +0000 UTC m=+1428.728003221" watchObservedRunningTime="2026-10-31 21:39:42.04113066 +0000 UTC m=+1428.734088055" Oct 31 21:39:45.080267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:45.079911 5188 patch_prober.go:29] interesting pod/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:39:45.080267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:45.079986 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:39:46.717914 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:46.717865 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w"] Oct 31 21:39:46.748548 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:46.748502 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls"] Oct 31 21:39:48.017063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:48.017029 5188 generic.go:356] "Generic (PLEG): container finished" podID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerID="e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe" exitCode=0 Oct 31 21:39:48.017538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:48.017102 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" event={"ID":"6255c981-759d-4a87-b6a1-dfb15b19fd6c","Type":"ContainerDied","Data":"e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe"} Oct 31 21:39:48.017585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:48.017555 5188 scope.go:122] "RemoveContainer" containerID="e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe" Oct 31 21:39:48.019452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:48.019422 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:39:48.019573 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:48.019509 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:39:48.019573 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:48.019544 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:39:48.033957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:48.033922 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff"] Oct 31 21:39:48.044109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:48.044080 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1420c1-fmjff"] Oct 31 21:39:49.031465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:49.031415 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" event={"ID":"6255c981-759d-4a87-b6a1-dfb15b19fd6c","Type":"ContainerStarted","Data":"90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89"} Oct 31 21:39:49.499569 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:49.499513 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca24a05-df1d-4c45-beda-9b8c9ae69779" path="/var/lib/kubelet/pods/0ca24a05-df1d-4c45-beda-9b8c9ae69779/volumes" Oct 31 21:39:49.835120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:49.835023 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:39:49.835368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:49.835337 5188 patch_prober.go:29] interesting pod/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:39:49.835436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:49.835388 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 21:39:50.835519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:50.835482 5188 patch_prober.go:29] interesting pod/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:39:50.835956 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:50.835538 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 21:39:51.718616 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:51.718544 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" podUID="bc4521b5-5894-44dc-9419-8cb97d8dcba1" containerName="payload-processing" containerID="cri-o://5f9250e4b2ed8b4f7fbb3f52262122504c9fe7ca6b5a9bdf92bb7fc6aeba5cff" gracePeriod=25 Oct 31 21:39:51.749406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:51.749285 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" podUID="e36c8273-fbb4-45dc-bec2-087f65da1f46" containerName="payload-pre-processing" containerID="cri-o://71060fef9edf3e13de11a0d9f42d7919a26c38c6da59b65a8e5bd454ad1a520c" gracePeriod=25 Oct 31 21:39:51.835158 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:51.835116 5188 patch_prober.go:29] interesting pod/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:39:51.835356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:51.835190 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 21:39:52.066910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.066866 5188 generic.go:356] "Generic (PLEG): container finished" podID="e36c8273-fbb4-45dc-bec2-087f65da1f46" containerID="71060fef9edf3e13de11a0d9f42d7919a26c38c6da59b65a8e5bd454ad1a520c" exitCode=0 Oct 31 21:39:52.067424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.067145 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" event={"ID":"e36c8273-fbb4-45dc-bec2-087f65da1f46","Type":"ContainerDied","Data":"71060fef9edf3e13de11a0d9f42d7919a26c38c6da59b65a8e5bd454ad1a520c"} Oct 31 21:39:52.071313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.071274 5188 generic.go:356] "Generic (PLEG): container finished" podID="bc4521b5-5894-44dc-9419-8cb97d8dcba1" containerID="5f9250e4b2ed8b4f7fbb3f52262122504c9fe7ca6b5a9bdf92bb7fc6aeba5cff" exitCode=0 Oct 31 21:39:52.071506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.071460 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" event={"ID":"bc4521b5-5894-44dc-9419-8cb97d8dcba1","Type":"ContainerDied","Data":"5f9250e4b2ed8b4f7fbb3f52262122504c9fe7ca6b5a9bdf92bb7fc6aeba5cff"} Oct 31 21:39:52.138142 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.138113 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:39:52.241983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.239788 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4521b5-5894-44dc-9419-8cb97d8dcba1-kube-api-access-82svc\" (UniqueName: \"kubernetes.io/projected/bc4521b5-5894-44dc-9419-8cb97d8dcba1-kube-api-access-82svc\") pod \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\" (UID: \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\") " Oct 31 21:39:52.241983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.239941 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4521b5-5894-44dc-9419-8cb97d8dcba1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4521b5-5894-44dc-9419-8cb97d8dcba1-plugins-config-volume\") pod \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\" (UID: \"bc4521b5-5894-44dc-9419-8cb97d8dcba1\") " Oct 31 21:39:52.241983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.241718 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4521b5-5894-44dc-9419-8cb97d8dcba1-plugins-config-volume" pod "bc4521b5-5894-44dc-9419-8cb97d8dcba1" (UID: "bc4521b5-5894-44dc-9419-8cb97d8dcba1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:39:52.243765 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.243726 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4521b5-5894-44dc-9419-8cb97d8dcba1-kube-api-access-82svc" pod "bc4521b5-5894-44dc-9419-8cb97d8dcba1" (UID: "bc4521b5-5894-44dc-9419-8cb97d8dcba1"). InnerVolumeSpecName "kube-api-access-82svc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:52.258018 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.257990 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:39:52.344959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.343883 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36c8273-fbb4-45dc-bec2-087f65da1f46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e36c8273-fbb4-45dc-bec2-087f65da1f46-plugins-config-volume\") pod \"e36c8273-fbb4-45dc-bec2-087f65da1f46\" (UID: \"e36c8273-fbb4-45dc-bec2-087f65da1f46\") " Oct 31 21:39:52.344959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.344098 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e36c8273-fbb4-45dc-bec2-087f65da1f46-kube-api-access-hjk9m\" (UniqueName: \"kubernetes.io/projected/e36c8273-fbb4-45dc-bec2-087f65da1f46-kube-api-access-hjk9m\") pod \"e36c8273-fbb4-45dc-bec2-087f65da1f46\" (UID: \"e36c8273-fbb4-45dc-bec2-087f65da1f46\") " Oct 31 21:39:52.344959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.344424 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36c8273-fbb4-45dc-bec2-087f65da1f46-plugins-config-volume" pod "e36c8273-fbb4-45dc-bec2-087f65da1f46" (UID: "e36c8273-fbb4-45dc-bec2-087f65da1f46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:39:52.344959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.344603 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc4521b5-5894-44dc-9419-8cb97d8dcba1-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:52.344959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.344623 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e36c8273-fbb4-45dc-bec2-087f65da1f46-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:52.344959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.344638 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82svc\" (UniqueName: \"kubernetes.io/projected/bc4521b5-5894-44dc-9419-8cb97d8dcba1-kube-api-access-82svc\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:52.347260 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.347233 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36c8273-fbb4-45dc-bec2-087f65da1f46-kube-api-access-hjk9m" pod "e36c8273-fbb4-45dc-bec2-087f65da1f46" (UID: "e36c8273-fbb4-45dc-bec2-087f65da1f46"). InnerVolumeSpecName "kube-api-access-hjk9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:52.446345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.446108 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjk9m\" (UniqueName: \"kubernetes.io/projected/e36c8273-fbb4-45dc-bec2-087f65da1f46-kube-api-access-hjk9m\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:39:52.461352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.460703 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx"] Oct 31 21:39:52.461837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.461809 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bc4521b5-5894-44dc-9419-8cb97d8dcba1" containerName="payload-processing" Oct 31 21:39:52.461964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.461838 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4521b5-5894-44dc-9419-8cb97d8dcba1" containerName="payload-processing" Oct 31 21:39:52.461964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.461884 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e36c8273-fbb4-45dc-bec2-087f65da1f46" containerName="payload-pre-processing" Oct 31 21:39:52.461964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.461893 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36c8273-fbb4-45dc-bec2-087f65da1f46" containerName="payload-pre-processing" Oct 31 21:39:52.462135 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.462120 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36c8273-fbb4-45dc-bec2-087f65da1f46" containerName="payload-pre-processing" Oct 31 21:39:52.462194 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.462153 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4521b5-5894-44dc-9419-8cb97d8dcba1" containerName="payload-processing" Oct 31 21:39:52.466362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.466293 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.509345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.503957 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9bbfc5-serving-cert\"" Oct 31 21:39:52.524875 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.515384 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx"] Oct 31 21:39:52.547621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.547568 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.547845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.547771 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvsbg\" (UniqueName: \"kubernetes.io/projected/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-kube-api-access-cvsbg\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.547845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.547821 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.548036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.548016 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-metrics-certs\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.649113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.649075 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.649352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.649166 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvsbg\" (UniqueName: \"kubernetes.io/projected/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-kube-api-access-cvsbg\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.649352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.649198 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.649352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.649342 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-metrics-certs\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.650166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.650022 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.652792 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.652759 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.653409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.653388 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-metrics-certs\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.670552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.670498 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvsbg\" (UniqueName: \"kubernetes.io/projected/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-kube-api-access-cvsbg\") pod \"maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.826479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.826433 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:52.835048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.835018 5188 patch_prober.go:29] interesting pod/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:39:52.835182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.835080 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 21:39:52.966051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.966017 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:39:52.966238 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:52.966098 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:39:53.122659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.122620 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" Oct 31 21:39:53.123162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.122625 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w" event={"ID":"bc4521b5-5894-44dc-9419-8cb97d8dcba1","Type":"ContainerDied","Data":"b21b933130dd736726645d9cbc27c241b4bbe7c9218bb41215ce107c7bdebb8d"} Oct 31 21:39:53.123162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.122764 5188 scope.go:122] "RemoveContainer" containerID="5f9250e4b2ed8b4f7fbb3f52262122504c9fe7ca6b5a9bdf92bb7fc6aeba5cff" Oct 31 21:39:53.127381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.126521 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" event={"ID":"e36c8273-fbb4-45dc-bec2-087f65da1f46","Type":"ContainerDied","Data":"cce932553667c63168531f21f88adf443a390fddfc24ae40463102f00a33ed99"} Oct 31 21:39:53.127381 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.126641 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls" Oct 31 21:39:53.158568 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.158539 5188 scope.go:122] "RemoveContainer" containerID="71060fef9edf3e13de11a0d9f42d7919a26c38c6da59b65a8e5bd454ad1a520c" Oct 31 21:39:53.214685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.214613 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx"] Oct 31 21:39:53.220435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.220257 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w"] Oct 31 21:39:53.227912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.227871 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-752623-57547865cc-lzn4w"] Oct 31 21:39:53.235096 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.235060 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls"] Oct 31 21:39:53.239024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.238997 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-85c7595ccf-jv2ls"] Oct 31 21:39:53.506409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.495687 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4521b5-5894-44dc-9419-8cb97d8dcba1" path="/var/lib/kubelet/pods/bc4521b5-5894-44dc-9419-8cb97d8dcba1/volumes" Oct 31 21:39:53.507464 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.507421 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e36c8273-fbb4-45dc-bec2-087f65da1f46" path="/var/lib/kubelet/pods/e36c8273-fbb4-45dc-bec2-087f65da1f46/volumes" Oct 31 21:39:53.835314 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.835284 5188 patch_prober.go:29] interesting pod/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:39:53.835485 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:53.835349 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Oct 31 21:39:54.137399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:54.137359 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" event={"ID":"e4400c6f-3abb-4ca9-bbfe-7e0fced98609","Type":"ContainerStarted","Data":"56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78"} Oct 31 21:39:54.137927 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:54.137435 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:39:54.137927 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:54.137460 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" event={"ID":"e4400c6f-3abb-4ca9-bbfe-7e0fced98609","Type":"ContainerStarted","Data":"294d6121e0df3d805e8c6b71f6cfaec39ed3857caa6655307c8725478869b3ee"} Oct 31 21:39:54.175722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:54.175649 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" podStartSLOduration=1.7255072359999999 podStartE2EDuration="2.175628572s" totalImagesPullingTime="450.121336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:39:54.174620124 +0000 UTC m=+1440.867577521" watchObservedRunningTime="2026-10-31 21:39:54.175628572 +0000 UTC m=+1440.868585965" Oct 31 21:39:54.839263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:54.839231 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:39:54.839636 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:54.839610 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:39:54.840505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:54.840482 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:39:55.352886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:55.352853 5188 scope.go:122] "RemoveContainer" containerID="41c21ad47e243bd4b7de8d53f12410a03c1d39a38b00ba5b870f154d93792e14" Oct 31 21:39:55.364285 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:55.364259 5188 scope.go:122] "RemoveContainer" containerID="50f6ca844c8cf38fa884365f3a40bb594326c1379efec5d18bcb6156cd1e92e9" Oct 31 21:39:55.378769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:55.378739 5188 scope.go:122] "RemoveContainer" containerID="e389c51ab92e1e8cab261d60fe53ad21bf1c4387a549a3680a9683febcd78eaa" Oct 31 21:39:55.392417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:39:55.392391 5188 scope.go:122] "RemoveContainer" containerID="4019666bb5bbd66d201bfb61e593e89c5ecd9e06b851f934c79caefdc237971f" Oct 31 21:40:00.173391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:00.172068 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:40:02.542809 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:40:02.542768 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:40:12.363639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:12.363603 5188 generic.go:356] "Generic (PLEG): container finished" podID="b0a9a747-3941-47bf-8765-fe70dd146648" containerID="929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6" exitCode=0 Oct 31 21:40:12.364140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:12.363674 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" event={"ID":"b0a9a747-3941-47bf-8765-fe70dd146648","Type":"ContainerDied","Data":"929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6"} Oct 31 21:40:12.364140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:12.364091 5188 scope.go:122] "RemoveContainer" containerID="929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6" Oct 31 21:40:12.365761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:12.365733 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:40:12.365825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:12.365807 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:40:12.365862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:12.365848 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:40:13.394371 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:13.393618 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" event={"ID":"b0a9a747-3941-47bf-8765-fe70dd146648","Type":"ContainerStarted","Data":"07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120"} Oct 31 21:40:14.175451 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:14.175416 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:40:14.183512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:14.183470 5188 patch_prober.go:29] interesting pod/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 21:40:14.183817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:14.183786 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 21:40:15.179458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:15.179418 5188 patch_prober.go:29] interesting pod/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 21:40:15.180123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:15.180086 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 21:40:15.341512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:15.341471 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:40:16.177313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:16.176824 5188 patch_prober.go:29] interesting pod/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 21:40:16.177313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:16.176896 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 21:40:16.916880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:16.916846 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6"] Oct 31 21:40:16.923523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:16.923492 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:16.924859 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:16.924823 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6"] Oct 31 21:40:16.960375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:16.952771 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-752623\"" Oct 31 21:40:16.960375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:16.952872 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-752623-dockercfg-4fd56\"" Oct 31 21:40:16.994349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:16.988961 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b209e1-d0bc-44fa-9e29-227b678d5e61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6\" (UID: \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:16.995135 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:16.994906 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rg48\" (UniqueName: \"kubernetes.io/projected/c9b209e1-d0bc-44fa-9e29-227b678d5e61-kube-api-access-7rg48\") pod \"payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6\" (UID: \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:17.117086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:17.117039 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rg48\" (UniqueName: \"kubernetes.io/projected/c9b209e1-d0bc-44fa-9e29-227b678d5e61-kube-api-access-7rg48\") pod \"payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6\" (UID: \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:17.133738 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:17.133696 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b209e1-d0bc-44fa-9e29-227b678d5e61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6\" (UID: \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:17.150358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:17.139230 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b209e1-d0bc-44fa-9e29-227b678d5e61-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6\" (UID: \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:17.249113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:17.249015 5188 patch_prober.go:29] interesting pod/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 21:40:17.250215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:17.250175 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 21:40:17.253928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:17.253893 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rg48\" (UniqueName: \"kubernetes.io/projected/c9b209e1-d0bc-44fa-9e29-227b678d5e61-kube-api-access-7rg48\") pod \"payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6\" (UID: \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:17.554512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:17.554404 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:17.910698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:17.910559 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6"] Oct 31 21:40:17.914003 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:40:17.913956 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b209e1_d0bc_44fa_9e29_227b678d5e61.slice/crio-d175c4f1b87fb6ef723e8e9f7a279d4bba7e0684410f46a2d22e794da77cd4ae WatchSource:0}: Error finding container d175c4f1b87fb6ef723e8e9f7a279d4bba7e0684410f46a2d22e794da77cd4ae: Status 404 returned error can't find the container with id d175c4f1b87fb6ef723e8e9f7a279d4bba7e0684410f46a2d22e794da77cd4ae Oct 31 21:40:18.209790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:18.204476 5188 patch_prober.go:29] interesting pod/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 21:40:18.209790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:18.204550 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 21:40:18.453850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:18.453807 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" event={"ID":"c9b209e1-d0bc-44fa-9e29-227b678d5e61","Type":"ContainerStarted","Data":"34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0"} Oct 31 21:40:18.454048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:18.453859 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" event={"ID":"c9b209e1-d0bc-44fa-9e29-227b678d5e61","Type":"ContainerStarted","Data":"d175c4f1b87fb6ef723e8e9f7a279d4bba7e0684410f46a2d22e794da77cd4ae"} Oct 31 21:40:18.454048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:18.453894 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:18.508426 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:18.505301 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" podStartSLOduration=2.505277037 podStartE2EDuration="2.505277037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:40:18.490255909 +0000 UTC m=+1465.183213302" watchObservedRunningTime="2026-10-31 21:40:18.505277037 +0000 UTC m=+1465.198234431" Oct 31 21:40:19.174842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:19.174792 5188 patch_prober.go:29] interesting pod/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 21:40:19.175079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:19.174864 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 21:40:20.181091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:20.181057 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:40:20.182466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:20.182442 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:40:29.463866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:29.463825 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:40:32.541834 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:40:32.541792 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:40:51.034030 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:51.033990 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd"] Oct 31 21:40:51.042578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:51.042544 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-607c7327-fgttd"] Oct 31 21:40:51.477232 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:51.477190 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074515e8-bd43-4184-a4a1-ad009a26e61d" path="/var/lib/kubelet/pods/074515e8-bd43-4184-a4a1-ad009a26e61d/volumes" Oct 31 21:40:53.794353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:53.794307 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:40:53.797081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:53.797049 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:40:55.497876 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:55.497823 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9"] Oct 31 21:40:55.498504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:55.498476 5188 scope.go:122] "RemoveContainer" containerID="27f0d8c47bc8155c77d315c0059b93d9efeb9c5a3fe95888baf17fab73058407" Oct 31 21:40:55.498816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:55.498778 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" podUID="74313ead-925b-4cc6-90f8-b91fff13c1c1" containerName="maas-api" containerID="cri-o://2cb3cf04151f65ca9d00155bea7e500e33224300ed37c943a1b3441fd26cc050" gracePeriod=30 Oct 31 21:40:55.547617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:55.547585 5188 scope.go:122] "RemoveContainer" containerID="0b935aa28384a22e385da0ffbc3526845d19574cc10268802ff7edb48d57a968" Oct 31 21:40:55.616995 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:55.616836 5188 scope.go:122] "RemoveContainer" containerID="d50039481a0edc1bd80cd4c5ee5e4623819f6aef53257afa4b2adc3714a644a7" Oct 31 21:40:55.842775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:55.842731 5188 generic.go:356] "Generic (PLEG): container finished" podID="74313ead-925b-4cc6-90f8-b91fff13c1c1" containerID="2cb3cf04151f65ca9d00155bea7e500e33224300ed37c943a1b3441fd26cc050" exitCode=0 Oct 31 21:40:55.843251 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:55.843222 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" event={"ID":"74313ead-925b-4cc6-90f8-b91fff13c1c1","Type":"ContainerDied","Data":"2cb3cf04151f65ca9d00155bea7e500e33224300ed37c943a1b3441fd26cc050"} Oct 31 21:40:55.879376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:55.875598 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:40:56.018637 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.018597 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74313ead-925b-4cc6-90f8-b91fff13c1c1-kube-api-access-p6k7s\" (UniqueName: \"kubernetes.io/projected/74313ead-925b-4cc6-90f8-b91fff13c1c1-kube-api-access-p6k7s\") pod \"74313ead-925b-4cc6-90f8-b91fff13c1c1\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " Oct 31 21:40:56.018855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.018751 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-maas-api-tls\") pod \"74313ead-925b-4cc6-90f8-b91fff13c1c1\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " Oct 31 21:40:56.018855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.018790 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74313ead-925b-4cc6-90f8-b91fff13c1c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74313ead-925b-4cc6-90f8-b91fff13c1c1-openshift-service-ca\") pod \"74313ead-925b-4cc6-90f8-b91fff13c1c1\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " Oct 31 21:40:56.018983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.018866 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-metrics-certs\") pod \"74313ead-925b-4cc6-90f8-b91fff13c1c1\" (UID: \"74313ead-925b-4cc6-90f8-b91fff13c1c1\") " Oct 31 21:40:56.020754 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.020724 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74313ead-925b-4cc6-90f8-b91fff13c1c1-openshift-service-ca" pod "74313ead-925b-4cc6-90f8-b91fff13c1c1" (UID: "74313ead-925b-4cc6-90f8-b91fff13c1c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:40:56.026415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.024990 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-maas-api-tls" pod "74313ead-925b-4cc6-90f8-b91fff13c1c1" (UID: "74313ead-925b-4cc6-90f8-b91fff13c1c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:40:56.029485 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.029455 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-metrics-certs" pod "74313ead-925b-4cc6-90f8-b91fff13c1c1" (UID: "74313ead-925b-4cc6-90f8-b91fff13c1c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:40:56.030197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.030158 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74313ead-925b-4cc6-90f8-b91fff13c1c1-kube-api-access-p6k7s" pod "74313ead-925b-4cc6-90f8-b91fff13c1c1" (UID: "74313ead-925b-4cc6-90f8-b91fff13c1c1"). InnerVolumeSpecName "kube-api-access-p6k7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:40:56.124497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.123888 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:40:56.124497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.123920 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74313ead-925b-4cc6-90f8-b91fff13c1c1-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:40:56.124497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.123930 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74313ead-925b-4cc6-90f8-b91fff13c1c1-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:40:56.124497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.123939 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6k7s\" (UniqueName: \"kubernetes.io/projected/74313ead-925b-4cc6-90f8-b91fff13c1c1-kube-api-access-p6k7s\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:40:56.852236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.852202 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" event={"ID":"74313ead-925b-4cc6-90f8-b91fff13c1c1","Type":"ContainerDied","Data":"09a93118c1ec50771ee9a65cd2a06728c1eed8a263018b2fe9db0c08459898c1"} Oct 31 21:40:56.852236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.852234 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" Oct 31 21:40:56.852776 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.852264 5188 scope.go:122] "RemoveContainer" containerID="2cb3cf04151f65ca9d00155bea7e500e33224300ed37c943a1b3441fd26cc050" Oct 31 21:40:56.899283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.899251 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9"] Oct 31 21:40:56.908494 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:56.908460 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9"] Oct 31 21:40:57.476222 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:57.476177 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74313ead-925b-4cc6-90f8-b91fff13c1c1" path="/var/lib/kubelet/pods/74313ead-925b-4cc6-90f8-b91fff13c1c1/volumes" Oct 31 21:40:58.800166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:40:58.800116 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-07afdd-64557ddf8-6pbg9" podUID="74313ead-925b-4cc6-90f8-b91fff13c1c1" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.158:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:41:02.539638 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:02.539594 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:41:04.149555 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.149515 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h"] Oct 31 21:41:04.150081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.149915 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" podUID="084b5c8c-54a8-4319-865d-3d1f43811abd" containerName="payload-processing" containerID="cri-o://df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2" gracePeriod=30 Oct 31 21:41:04.170717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.170683 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk"] Oct 31 21:41:04.170977 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.170944 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" podUID="fa027455-c262-4144-9928-d9c0c43fd335" containerName="payload-pre-processing" containerID="cri-o://6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039" gracePeriod=30 Oct 31 21:41:04.617801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.617486 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:41:04.619810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.619782 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:41:04.646244 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.646199 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l"] Oct 31 21:41:04.647255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.647231 5188 cpu_manager.go:414] "containerMap: removing container" podUID="74313ead-925b-4cc6-90f8-b91fff13c1c1" containerName="maas-api" Oct 31 21:41:04.647255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.647256 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74313ead-925b-4cc6-90f8-b91fff13c1c1" containerName="maas-api" Oct 31 21:41:04.647458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.647302 5188 cpu_manager.go:414] "containerMap: removing container" podUID="fa027455-c262-4144-9928-d9c0c43fd335" containerName="payload-pre-processing" Oct 31 21:41:04.647458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.647310 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa027455-c262-4144-9928-d9c0c43fd335" containerName="payload-pre-processing" Oct 31 21:41:04.647458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.647355 5188 cpu_manager.go:414] "containerMap: removing container" podUID="084b5c8c-54a8-4319-865d-3d1f43811abd" containerName="payload-processing" Oct 31 21:41:04.647458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.647364 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084b5c8c-54a8-4319-865d-3d1f43811abd" containerName="payload-processing" Oct 31 21:41:04.647674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.647506 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa027455-c262-4144-9928-d9c0c43fd335" containerName="payload-pre-processing" Oct 31 21:41:04.647674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.647519 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="084b5c8c-54a8-4319-865d-3d1f43811abd" containerName="payload-processing" Oct 31 21:41:04.647674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.647536 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="74313ead-925b-4cc6-90f8-b91fff13c1c1" containerName="maas-api" Oct 31 21:41:04.652072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.652042 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.689699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.689654 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa027455-c262-4144-9928-d9c0c43fd335-kube-api-access-9q7zd\" (UniqueName: \"kubernetes.io/projected/fa027455-c262-4144-9928-d9c0c43fd335-kube-api-access-9q7zd\") pod \"fa027455-c262-4144-9928-d9c0c43fd335\" (UID: \"fa027455-c262-4144-9928-d9c0c43fd335\") " Oct 31 21:41:04.689911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.689766 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084b5c8c-54a8-4319-865d-3d1f43811abd-kube-api-access-6pdhf\" (UniqueName: \"kubernetes.io/projected/084b5c8c-54a8-4319-865d-3d1f43811abd-kube-api-access-6pdhf\") pod \"084b5c8c-54a8-4319-865d-3d1f43811abd\" (UID: \"084b5c8c-54a8-4319-865d-3d1f43811abd\") " Oct 31 21:41:04.689911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.689808 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084b5c8c-54a8-4319-865d-3d1f43811abd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084b5c8c-54a8-4319-865d-3d1f43811abd-plugins-config-volume\") pod \"084b5c8c-54a8-4319-865d-3d1f43811abd\" (UID: \"084b5c8c-54a8-4319-865d-3d1f43811abd\") " Oct 31 21:41:04.690038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.689965 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa027455-c262-4144-9928-d9c0c43fd335-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa027455-c262-4144-9928-d9c0c43fd335-plugins-config-volume\") pod \"fa027455-c262-4144-9928-d9c0c43fd335\" (UID: \"fa027455-c262-4144-9928-d9c0c43fd335\") " Oct 31 21:41:04.690753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.690215 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mh6\" (UniqueName: \"kubernetes.io/projected/f45bf814-a94a-4bdf-bbf4-00591f7538a4-kube-api-access-d8mh6\") pod \"maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.691666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.691010 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.691666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.691128 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.692556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.692241 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084b5c8c-54a8-4319-865d-3d1f43811abd-plugins-config-volume" pod "084b5c8c-54a8-4319-865d-3d1f43811abd" (UID: "084b5c8c-54a8-4319-865d-3d1f43811abd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:04.693298 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.692838 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa027455-c262-4144-9928-d9c0c43fd335-plugins-config-volume" pod "fa027455-c262-4144-9928-d9c0c43fd335" (UID: "fa027455-c262-4144-9928-d9c0c43fd335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:04.704962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.704885 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:41:04.704962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.704962 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:41:04.705494 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.705465 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084b5c8c-54a8-4319-865d-3d1f43811abd-kube-api-access-6pdhf" pod "084b5c8c-54a8-4319-865d-3d1f43811abd" (UID: "084b5c8c-54a8-4319-865d-3d1f43811abd"). InnerVolumeSpecName "kube-api-access-6pdhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:04.706964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.706938 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l"] Oct 31 21:41:04.709845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.709814 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa027455-c262-4144-9928-d9c0c43fd335-kube-api-access-9q7zd" pod "fa027455-c262-4144-9928-d9c0c43fd335" (UID: "fa027455-c262-4144-9928-d9c0c43fd335"). InnerVolumeSpecName "kube-api-access-9q7zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:04.793096 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.793049 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mh6\" (UniqueName: \"kubernetes.io/projected/f45bf814-a94a-4bdf-bbf4-00591f7538a4-kube-api-access-d8mh6\") pod \"maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.793311 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.793247 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.793465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.793371 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.793531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.793462 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q7zd\" (UniqueName: \"kubernetes.io/projected/fa027455-c262-4144-9928-d9c0c43fd335-kube-api-access-9q7zd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:04.793531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.793479 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pdhf\" (UniqueName: \"kubernetes.io/projected/084b5c8c-54a8-4319-865d-3d1f43811abd-kube-api-access-6pdhf\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:04.793531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.793494 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/084b5c8c-54a8-4319-865d-3d1f43811abd-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:04.793531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.793509 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa027455-c262-4144-9928-d9c0c43fd335-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:04.794002 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.793975 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.796257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.796225 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.814163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.814124 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mh6\" (UniqueName: \"kubernetes.io/projected/f45bf814-a94a-4bdf-bbf4-00591f7538a4-kube-api-access-d8mh6\") pod \"maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:04.954153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.954064 5188 generic.go:356] "Generic (PLEG): container finished" podID="084b5c8c-54a8-4319-865d-3d1f43811abd" containerID="df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2" exitCode=0 Oct 31 21:41:04.954153 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.954128 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" Oct 31 21:41:04.954465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.954151 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" event={"ID":"084b5c8c-54a8-4319-865d-3d1f43811abd","Type":"ContainerDied","Data":"df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2"} Oct 31 21:41:04.954465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.954208 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h" event={"ID":"084b5c8c-54a8-4319-865d-3d1f43811abd","Type":"ContainerDied","Data":"7955ee1c27657922b70d1906890dae6bc0da49e2f75bf010f146e7c3ead0e048"} Oct 31 21:41:04.954465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.954240 5188 scope.go:122] "RemoveContainer" containerID="df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2" Oct 31 21:41:04.955753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.955725 5188 generic.go:356] "Generic (PLEG): container finished" podID="fa027455-c262-4144-9928-d9c0c43fd335" containerID="6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039" exitCode=0 Oct 31 21:41:04.955870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.955798 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" Oct 31 21:41:04.955870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.955810 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" event={"ID":"fa027455-c262-4144-9928-d9c0c43fd335","Type":"ContainerDied","Data":"6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039"} Oct 31 21:41:04.955969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.955866 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk" event={"ID":"fa027455-c262-4144-9928-d9c0c43fd335","Type":"ContainerDied","Data":"49b001e0247383253c88eedd0b9d31ef69f641a090eff0ad90a8387a47eb1ac2"} Oct 31 21:41:04.968820 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.968798 5188 scope.go:122] "RemoveContainer" containerID="df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2" Oct 31 21:41:04.969181 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:04.969158 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2\": container with ID starting with df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2 not found: ID does not exist" containerID="df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2" Oct 31 21:41:04.969292 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.969192 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2"} err="failed to get container status \"df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2\": rpc error: code = NotFound desc = could not find container \"df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2\": container with ID starting with df1474d79dfc8b807d013b432a6cb375c2810ef4714c8799c4c1c758651c06a2 not found: ID does not exist" Oct 31 21:41:04.969292 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.969218 5188 scope.go:122] "RemoveContainer" containerID="6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039" Oct 31 21:41:04.982679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.982572 5188 scope.go:122] "RemoveContainer" containerID="6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039" Oct 31 21:41:04.982973 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:04.982952 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039\": container with ID starting with 6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039 not found: ID does not exist" containerID="6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039" Oct 31 21:41:04.983051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:04.982981 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039"} err="failed to get container status \"6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039\": rpc error: code = NotFound desc = could not find container \"6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039\": container with ID starting with 6e082221a4bf84f0adcd10ca7eb9a4812d32054aade4cd66f744db5a3dd38039 not found: ID does not exist" Oct 31 21:41:05.003699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.003676 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h"] Oct 31 21:41:05.008069 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.008043 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-89d4a2-7b46f48b7d-q8n2h"] Oct 31 21:41:05.010741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.010718 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:05.043291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.038247 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk"] Oct 31 21:41:05.043291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.043240 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-89d4a2-6d4b4fb4f5-6jkqk"] Oct 31 21:41:05.322905 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:41:05.322872 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45bf814_a94a_4bdf_bbf4_00591f7538a4.slice/crio-88c9ff4d735ae11d7c75f441e64f599e5faa8789521349528b344c6942c3940a WatchSource:0}: Error finding container 88c9ff4d735ae11d7c75f441e64f599e5faa8789521349528b344c6942c3940a: Status 404 returned error can't find the container with id 88c9ff4d735ae11d7c75f441e64f599e5faa8789521349528b344c6942c3940a Oct 31 21:41:05.323291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.323175 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l"] Oct 31 21:41:05.476295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.476255 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084b5c8c-54a8-4319-865d-3d1f43811abd" path="/var/lib/kubelet/pods/084b5c8c-54a8-4319-865d-3d1f43811abd/volumes" Oct 31 21:41:05.476855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.476833 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa027455-c262-4144-9928-d9c0c43fd335" path="/var/lib/kubelet/pods/fa027455-c262-4144-9928-d9c0c43fd335/volumes" Oct 31 21:41:05.969217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.969122 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" event={"ID":"f45bf814-a94a-4bdf-bbf4-00591f7538a4","Type":"ContainerStarted","Data":"4cbedf6a214bb6c4dc9af901b097b6468372fc5fb92cfc321e47918cbed24712"} Oct 31 21:41:05.969217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:05.969177 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" event={"ID":"f45bf814-a94a-4bdf-bbf4-00591f7538a4","Type":"ContainerStarted","Data":"88c9ff4d735ae11d7c75f441e64f599e5faa8789521349528b344c6942c3940a"} Oct 31 21:41:06.010904 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:06.010836 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" podStartSLOduration=2.010816787 podStartE2EDuration="2.010816787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:41:06.004139482 +0000 UTC m=+1512.697096875" watchObservedRunningTime="2026-10-31 21:41:06.010816787 +0000 UTC m=+1512.703774179" Oct 31 21:41:06.978125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:06.978087 5188 generic.go:356] "Generic (PLEG): container finished" podID="f45bf814-a94a-4bdf-bbf4-00591f7538a4" containerID="4cbedf6a214bb6c4dc9af901b097b6468372fc5fb92cfc321e47918cbed24712" exitCode=0 Oct 31 21:41:06.978608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:06.978122 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" event={"ID":"f45bf814-a94a-4bdf-bbf4-00591f7538a4","Type":"ContainerDied","Data":"4cbedf6a214bb6c4dc9af901b097b6468372fc5fb92cfc321e47918cbed24712"} Oct 31 21:41:08.160090 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.160063 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:08.271369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.271252 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-openshift-ca\") pod \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " Oct 31 21:41:08.271573 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.271385 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45bf814-a94a-4bdf-bbf4-00591f7538a4-kube-api-access-d8mh6\" (UniqueName: \"kubernetes.io/projected/f45bf814-a94a-4bdf-bbf4-00591f7538a4-kube-api-access-d8mh6\") pod \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " Oct 31 21:41:08.271573 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.271509 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-xks-ca\") pod \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\" (UID: \"f45bf814-a94a-4bdf-bbf4-00591f7538a4\") " Oct 31 21:41:08.271747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.271717 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-openshift-ca" pod "f45bf814-a94a-4bdf-bbf4-00591f7538a4" (UID: "f45bf814-a94a-4bdf-bbf4-00591f7538a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:08.272211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.272187 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:08.273673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.273642 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-xks-ca" pod "f45bf814-a94a-4bdf-bbf4-00591f7538a4" (UID: "f45bf814-a94a-4bdf-bbf4-00591f7538a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:08.273806 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.273757 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45bf814-a94a-4bdf-bbf4-00591f7538a4-kube-api-access-d8mh6" pod "f45bf814-a94a-4bdf-bbf4-00591f7538a4" (UID: "f45bf814-a94a-4bdf-bbf4-00591f7538a4"). InnerVolumeSpecName "kube-api-access-d8mh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:08.373207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.373174 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8mh6\" (UniqueName: \"kubernetes.io/projected/f45bf814-a94a-4bdf-bbf4-00591f7538a4-kube-api-access-d8mh6\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:08.373207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.373204 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45bf814-a94a-4bdf-bbf4-00591f7538a4-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:08.997669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.997637 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" Oct 31 21:41:08.997852 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.997665 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l" event={"ID":"f45bf814-a94a-4bdf-bbf4-00591f7538a4","Type":"ContainerDied","Data":"88c9ff4d735ae11d7c75f441e64f599e5faa8789521349528b344c6942c3940a"} Oct 31 21:41:08.997852 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:08.997719 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88c9ff4d735ae11d7c75f441e64f599e5faa8789521349528b344c6942c3940a" Oct 31 21:41:11.199496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.199452 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm"] Oct 31 21:41:11.200590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.200567 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f45bf814-a94a-4bdf-bbf4-00591f7538a4" containerName="revoke-keys" Oct 31 21:41:11.200649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.200593 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45bf814-a94a-4bdf-bbf4-00591f7538a4" containerName="revoke-keys" Oct 31 21:41:11.200771 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.200759 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45bf814-a94a-4bdf-bbf4-00591f7538a4" containerName="revoke-keys" Oct 31 21:41:11.205298 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.205274 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.222879 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.222847 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm"] Oct 31 21:41:11.226023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.225997 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:41:11.226144 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.226130 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:41:11.315836 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.315787 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ba032-ac87-408b-807f-aad53b917137-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.316036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.316013 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba032-ac87-408b-807f-aad53b917137-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.316205 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.316169 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqrzx\" (UniqueName: \"kubernetes.io/projected/e12ba032-ac87-408b-807f-aad53b917137-kube-api-access-nqrzx\") pod \"maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.417056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.417009 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ba032-ac87-408b-807f-aad53b917137-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.417275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.417195 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba032-ac87-408b-807f-aad53b917137-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.417390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.417315 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqrzx\" (UniqueName: \"kubernetes.io/projected/e12ba032-ac87-408b-807f-aad53b917137-kube-api-access-nqrzx\") pod \"maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.418092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.418064 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba032-ac87-408b-807f-aad53b917137-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.419547 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.419527 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ba032-ac87-408b-807f-aad53b917137-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.438832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.438799 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqrzx\" (UniqueName: \"kubernetes.io/projected/e12ba032-ac87-408b-807f-aad53b917137-kube-api-access-nqrzx\") pod \"maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.525483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.525394 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:11.841181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:11.841148 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm"] Oct 31 21:41:11.841800 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:41:11.841768 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12ba032_ac87_408b_807f_aad53b917137.slice/crio-bb9d7646c0d48aa8c83000df9a1a77c714a29219d0ccdd1b3905af063c14dd45 WatchSource:0}: Error finding container bb9d7646c0d48aa8c83000df9a1a77c714a29219d0ccdd1b3905af063c14dd45: Status 404 returned error can't find the container with id bb9d7646c0d48aa8c83000df9a1a77c714a29219d0ccdd1b3905af063c14dd45 Oct 31 21:41:12.029280 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:12.029240 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" event={"ID":"e12ba032-ac87-408b-807f-aad53b917137","Type":"ContainerStarted","Data":"bb9d7646c0d48aa8c83000df9a1a77c714a29219d0ccdd1b3905af063c14dd45"} Oct 31 21:41:13.040036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:13.039997 5188 generic.go:356] "Generic (PLEG): container finished" podID="e12ba032-ac87-408b-807f-aad53b917137" containerID="720f8bf66e7823ae710309120d806994bcb46eb1783106a7732b40f351e05b46" exitCode=0 Oct 31 21:41:13.040580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:13.040246 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" event={"ID":"e12ba032-ac87-408b-807f-aad53b917137","Type":"ContainerDied","Data":"720f8bf66e7823ae710309120d806994bcb46eb1783106a7732b40f351e05b46"} Oct 31 21:41:14.221892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.221860 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:14.364695 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.364638 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12ba032-ac87-408b-807f-aad53b917137-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ba032-ac87-408b-807f-aad53b917137-maas-api-xks-ca\") pod \"e12ba032-ac87-408b-807f-aad53b917137\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " Oct 31 21:41:14.364912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.364898 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12ba032-ac87-408b-807f-aad53b917137-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba032-ac87-408b-807f-aad53b917137-maas-api-openshift-ca\") pod \"e12ba032-ac87-408b-807f-aad53b917137\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " Oct 31 21:41:14.364991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.364974 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12ba032-ac87-408b-807f-aad53b917137-kube-api-access-nqrzx\" (UniqueName: \"kubernetes.io/projected/e12ba032-ac87-408b-807f-aad53b917137-kube-api-access-nqrzx\") pod \"e12ba032-ac87-408b-807f-aad53b917137\" (UID: \"e12ba032-ac87-408b-807f-aad53b917137\") " Oct 31 21:41:14.365252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.365225 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12ba032-ac87-408b-807f-aad53b917137-maas-api-openshift-ca" pod "e12ba032-ac87-408b-807f-aad53b917137" (UID: "e12ba032-ac87-408b-807f-aad53b917137"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:14.365465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.365446 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba032-ac87-408b-807f-aad53b917137-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:14.366897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.366871 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12ba032-ac87-408b-807f-aad53b917137-maas-api-xks-ca" pod "e12ba032-ac87-408b-807f-aad53b917137" (UID: "e12ba032-ac87-408b-807f-aad53b917137"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:14.367123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.367101 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12ba032-ac87-408b-807f-aad53b917137-kube-api-access-nqrzx" pod "e12ba032-ac87-408b-807f-aad53b917137" (UID: "e12ba032-ac87-408b-807f-aad53b917137"). InnerVolumeSpecName "kube-api-access-nqrzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:14.466575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.466537 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e12ba032-ac87-408b-807f-aad53b917137-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:14.466575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:14.466568 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqrzx\" (UniqueName: \"kubernetes.io/projected/e12ba032-ac87-408b-807f-aad53b917137-kube-api-access-nqrzx\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:15.059516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:15.059484 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" Oct 31 21:41:15.059713 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:15.059513 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm" event={"ID":"e12ba032-ac87-408b-807f-aad53b917137","Type":"ContainerDied","Data":"bb9d7646c0d48aa8c83000df9a1a77c714a29219d0ccdd1b3905af063c14dd45"} Oct 31 21:41:15.059713 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:15.059577 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb9d7646c0d48aa8c83000df9a1a77c714a29219d0ccdd1b3905af063c14dd45" Oct 31 21:41:15.160078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:15.159850 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj"] Oct 31 21:41:15.209780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:15.209735 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs"] Oct 31 21:41:20.160667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.160613 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" podUID="ed05370e-a7eb-41e7-ab40-2bd3deba1f9d" containerName="payload-processing" containerID="cri-o://ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5" gracePeriod=25 Oct 31 21:41:20.211173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.211117 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" podUID="f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a" containerName="payload-pre-processing" containerID="cri-o://0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8" gracePeriod=25 Oct 31 21:41:20.535653 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.535602 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:41:20.599452 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.599415 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:41:20.666338 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.666286 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-plugins-config-volume\") pod \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\" (UID: \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\") " Oct 31 21:41:20.666549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.666454 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-plugins-config-volume\") pod \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\" (UID: \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\") " Oct 31 21:41:20.666549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.666505 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-kube-api-access-6t56q\" (UniqueName: \"kubernetes.io/projected/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-kube-api-access-6t56q\") pod \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\" (UID: \"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a\") " Oct 31 21:41:20.666685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.666585 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-kube-api-access-qqznc\" (UniqueName: \"kubernetes.io/projected/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-kube-api-access-qqznc\") pod \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\" (UID: \"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d\") " Oct 31 21:41:20.666861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.666742 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-plugins-config-volume" pod "ed05370e-a7eb-41e7-ab40-2bd3deba1f9d" (UID: "ed05370e-a7eb-41e7-ab40-2bd3deba1f9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:20.667058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.667032 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-plugins-config-volume" pod "f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a" (UID: "f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:20.667058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.667062 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:20.669116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.669084 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-kube-api-access-6t56q" pod "f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a" (UID: "f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a"). InnerVolumeSpecName "kube-api-access-6t56q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:20.669234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.669172 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-kube-api-access-qqznc" pod "ed05370e-a7eb-41e7-ab40-2bd3deba1f9d" (UID: "ed05370e-a7eb-41e7-ab40-2bd3deba1f9d"). InnerVolumeSpecName "kube-api-access-qqznc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:20.769046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.768963 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqznc\" (UniqueName: \"kubernetes.io/projected/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d-kube-api-access-qqznc\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:20.769046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.768998 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:20.769046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:20.769011 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t56q\" (UniqueName: \"kubernetes.io/projected/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a-kube-api-access-6t56q\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:21.112269 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.112229 5188 generic.go:356] "Generic (PLEG): container finished" podID="f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a" containerID="0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8" exitCode=0 Oct 31 21:41:21.112479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.112293 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" Oct 31 21:41:21.112479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.112358 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" event={"ID":"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a","Type":"ContainerDied","Data":"0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8"} Oct 31 21:41:21.112479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.112429 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs" event={"ID":"f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a","Type":"ContainerDied","Data":"39ddf6f43bd6d755311e39cead88a7d2d9e23f4ea2c6f6bfb05a45bf06edb5e7"} Oct 31 21:41:21.112479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.112460 5188 scope.go:122] "RemoveContainer" containerID="0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8" Oct 31 21:41:21.114750 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.114722 5188 generic.go:356] "Generic (PLEG): container finished" podID="ed05370e-a7eb-41e7-ab40-2bd3deba1f9d" containerID="ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5" exitCode=0 Oct 31 21:41:21.114891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.114788 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" Oct 31 21:41:21.114891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.114792 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" event={"ID":"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d","Type":"ContainerDied","Data":"ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5"} Oct 31 21:41:21.115013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.114900 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj" event={"ID":"ed05370e-a7eb-41e7-ab40-2bd3deba1f9d","Type":"ContainerDied","Data":"a95f23366023739921008c7984f8960f889e39ebe227cd55e2b54e3387e6c965"} Oct 31 21:41:21.127796 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.127711 5188 scope.go:122] "RemoveContainer" containerID="0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8" Oct 31 21:41:21.128230 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:21.128200 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8\": container with ID starting with 0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8 not found: ID does not exist" containerID="0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8" Oct 31 21:41:21.128302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.128244 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8"} err="failed to get container status \"0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8\": rpc error: code = NotFound desc = could not find container \"0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8\": container with ID starting with 0099e0253e9be61fd3cec19227c4e65403014aa6818ad5b9dbb0029e8ef399f8 not found: ID does not exist" Oct 31 21:41:21.128302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.128270 5188 scope.go:122] "RemoveContainer" containerID="ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5" Oct 31 21:41:21.140721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.140696 5188 scope.go:122] "RemoveContainer" containerID="ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5" Oct 31 21:41:21.141062 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:21.141043 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5\": container with ID starting with ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5 not found: ID does not exist" containerID="ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5" Oct 31 21:41:21.141158 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.141068 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5"} err="failed to get container status \"ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5\": rpc error: code = NotFound desc = could not find container \"ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5\": container with ID starting with ef488bab680b79470a3f4de7a607b0bc9b182830653653c730ecf69ec4b5c1a5 not found: ID does not exist" Oct 31 21:41:21.165720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.165685 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs"] Oct 31 21:41:21.170515 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.170480 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3dae1-b68556cb7-nrmjs"] Oct 31 21:41:21.215145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.215110 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj"] Oct 31 21:41:21.221591 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.221552 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3dae1-ccf495dff-kl5cj"] Oct 31 21:41:21.474529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.474444 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed05370e-a7eb-41e7-ab40-2bd3deba1f9d" path="/var/lib/kubelet/pods/ed05370e-a7eb-41e7-ab40-2bd3deba1f9d/volumes" Oct 31 21:41:21.474904 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:21.474888 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a" path="/var/lib/kubelet/pods/f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a/volumes" Oct 31 21:41:28.172170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.172136 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v"] Oct 31 21:41:28.173349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.172407 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" podUID="892582bc-6881-4e05-be5d-1fc72dc82847" containerName="payload-processing" containerID="cri-o://70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f" gracePeriod=30 Oct 31 21:41:28.218774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.218730 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx"] Oct 31 21:41:28.219227 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.219191 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" podUID="2d2107fc-814c-4a1f-adfa-e2023e8c52bf" containerName="payload-pre-processing" containerID="cri-o://10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d" gracePeriod=30 Oct 31 21:41:28.694346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.686505 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:41:28.722703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.722676 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:41:28.791633 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.791593 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-kube-api-access-4wjmd\" (UniqueName: \"kubernetes.io/projected/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-kube-api-access-4wjmd\") pod \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\" (UID: \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\") " Oct 31 21:41:28.791844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.791661 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892582bc-6881-4e05-be5d-1fc72dc82847-kube-api-access-zr4rv\" (UniqueName: \"kubernetes.io/projected/892582bc-6881-4e05-be5d-1fc72dc82847-kube-api-access-zr4rv\") pod \"892582bc-6881-4e05-be5d-1fc72dc82847\" (UID: \"892582bc-6881-4e05-be5d-1fc72dc82847\") " Oct 31 21:41:28.791844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.791779 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892582bc-6881-4e05-be5d-1fc72dc82847-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892582bc-6881-4e05-be5d-1fc72dc82847-plugins-config-volume\") pod \"892582bc-6881-4e05-be5d-1fc72dc82847\" (UID: \"892582bc-6881-4e05-be5d-1fc72dc82847\") " Oct 31 21:41:28.791982 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.791924 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-plugins-config-volume\") pod \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\" (UID: \"2d2107fc-814c-4a1f-adfa-e2023e8c52bf\") " Oct 31 21:41:28.792392 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.792358 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892582bc-6881-4e05-be5d-1fc72dc82847-plugins-config-volume" pod "892582bc-6881-4e05-be5d-1fc72dc82847" (UID: "892582bc-6881-4e05-be5d-1fc72dc82847"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:28.792531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.792506 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-plugins-config-volume" pod "2d2107fc-814c-4a1f-adfa-e2023e8c52bf" (UID: "2d2107fc-814c-4a1f-adfa-e2023e8c52bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:28.792694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.792674 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/892582bc-6881-4e05-be5d-1fc72dc82847-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:28.792757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.792700 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:28.794708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.794682 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-kube-api-access-4wjmd" pod "2d2107fc-814c-4a1f-adfa-e2023e8c52bf" (UID: "2d2107fc-814c-4a1f-adfa-e2023e8c52bf"). InnerVolumeSpecName "kube-api-access-4wjmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:28.795006 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.794889 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892582bc-6881-4e05-be5d-1fc72dc82847-kube-api-access-zr4rv" pod "892582bc-6881-4e05-be5d-1fc72dc82847" (UID: "892582bc-6881-4e05-be5d-1fc72dc82847"). InnerVolumeSpecName "kube-api-access-zr4rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:28.851123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.851039 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm"] Oct 31 21:41:28.854935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.854900 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c40fc3-s2bmm"] Oct 31 21:41:28.894447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.894404 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wjmd\" (UniqueName: \"kubernetes.io/projected/2d2107fc-814c-4a1f-adfa-e2023e8c52bf-kube-api-access-4wjmd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:28.894447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:28.894440 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr4rv\" (UniqueName: \"kubernetes.io/projected/892582bc-6881-4e05-be5d-1fc72dc82847-kube-api-access-zr4rv\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:29.185200 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.185165 5188 generic.go:356] "Generic (PLEG): container finished" podID="892582bc-6881-4e05-be5d-1fc72dc82847" containerID="70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f" exitCode=0 Oct 31 21:41:29.185718 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.185228 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" Oct 31 21:41:29.185718 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.185254 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" event={"ID":"892582bc-6881-4e05-be5d-1fc72dc82847","Type":"ContainerDied","Data":"70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f"} Oct 31 21:41:29.185718 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.185332 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v" event={"ID":"892582bc-6881-4e05-be5d-1fc72dc82847","Type":"ContainerDied","Data":"e2a782287da7c6653d4cd8093467194b64aa2560067db65cb7aa20ec850fa643"} Oct 31 21:41:29.185718 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.185363 5188 scope.go:122] "RemoveContainer" containerID="70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f" Oct 31 21:41:29.187316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.187132 5188 generic.go:356] "Generic (PLEG): container finished" podID="2d2107fc-814c-4a1f-adfa-e2023e8c52bf" containerID="10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d" exitCode=0 Oct 31 21:41:29.187316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.187236 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" Oct 31 21:41:29.187493 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.187316 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" event={"ID":"2d2107fc-814c-4a1f-adfa-e2023e8c52bf","Type":"ContainerDied","Data":"10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d"} Oct 31 21:41:29.187493 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.187399 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx" event={"ID":"2d2107fc-814c-4a1f-adfa-e2023e8c52bf","Type":"ContainerDied","Data":"4a77416df745f9a520bd9937cef258c0dab03c06ff014c5f15a9568e908f71e1"} Oct 31 21:41:29.218289 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.218258 5188 scope.go:122] "RemoveContainer" containerID="70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f" Oct 31 21:41:29.218939 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:29.218914 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f\": container with ID starting with 70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f not found: ID does not exist" containerID="70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f" Oct 31 21:41:29.219038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.218947 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f"} err="failed to get container status \"70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f\": rpc error: code = NotFound desc = could not find container \"70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f\": container with ID starting with 70dc33f6a8887f61e957f4b0e66b893df6582a2a4a07dd83a26a6a6ff56bc54f not found: ID does not exist" Oct 31 21:41:29.219038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.218966 5188 scope.go:122] "RemoveContainer" containerID="10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d" Oct 31 21:41:29.230287 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.230264 5188 scope.go:122] "RemoveContainer" containerID="10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d" Oct 31 21:41:29.230681 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:29.230655 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d\": container with ID starting with 10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d not found: ID does not exist" containerID="10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d" Oct 31 21:41:29.230755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.230691 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d"} err="failed to get container status \"10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d\": rpc error: code = NotFound desc = could not find container \"10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d\": container with ID starting with 10f0f2a7295e75f709898140fd021f7e20665e976c1854d9ffa0f0a5ef4add2d not found: ID does not exist" Oct 31 21:41:29.244733 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.244701 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v"] Oct 31 21:41:29.250914 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.250881 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c40fc3-5dd57656d8-s2x4v"] Oct 31 21:41:29.280506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.280465 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx"] Oct 31 21:41:29.284831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.284802 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c40fc3-5c799cbb5f-s7vvx"] Oct 31 21:41:29.475508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.475426 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2107fc-814c-4a1f-adfa-e2023e8c52bf" path="/var/lib/kubelet/pods/2d2107fc-814c-4a1f-adfa-e2023e8c52bf/volumes" Oct 31 21:41:29.475976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.475958 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892582bc-6881-4e05-be5d-1fc72dc82847" path="/var/lib/kubelet/pods/892582bc-6881-4e05-be5d-1fc72dc82847/volumes" Oct 31 21:41:29.476463 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:29.476446 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12ba032-ac87-408b-807f-aad53b917137" path="/var/lib/kubelet/pods/e12ba032-ac87-408b-807f-aad53b917137/volumes" Oct 31 21:41:30.593627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.593587 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c"] Oct 31 21:41:30.594286 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594269 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ed05370e-a7eb-41e7-ab40-2bd3deba1f9d" containerName="payload-processing" Oct 31 21:41:30.594367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594287 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed05370e-a7eb-41e7-ab40-2bd3deba1f9d" containerName="payload-processing" Oct 31 21:41:30.594367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594305 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e12ba032-ac87-408b-807f-aad53b917137" containerName="revoke-keys" Oct 31 21:41:30.594367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594310 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12ba032-ac87-408b-807f-aad53b917137" containerName="revoke-keys" Oct 31 21:41:30.594367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594336 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a" containerName="payload-pre-processing" Oct 31 21:41:30.594367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594342 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a" containerName="payload-pre-processing" Oct 31 21:41:30.594537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594372 5188 cpu_manager.go:414] "containerMap: removing container" podUID="2d2107fc-814c-4a1f-adfa-e2023e8c52bf" containerName="payload-pre-processing" Oct 31 21:41:30.594537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594377 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2107fc-814c-4a1f-adfa-e2023e8c52bf" containerName="payload-pre-processing" Oct 31 21:41:30.594537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594386 5188 cpu_manager.go:414] "containerMap: removing container" podUID="892582bc-6881-4e05-be5d-1fc72dc82847" containerName="payload-processing" Oct 31 21:41:30.594537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594390 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892582bc-6881-4e05-be5d-1fc72dc82847" containerName="payload-processing" Oct 31 21:41:30.594537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594486 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed05370e-a7eb-41e7-ab40-2bd3deba1f9d" containerName="payload-processing" Oct 31 21:41:30.594537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594497 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12ba032-ac87-408b-807f-aad53b917137" containerName="revoke-keys" Oct 31 21:41:30.594537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594503 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f49e1e-4677-49e1-9bcf-f5fe9ceea30a" containerName="payload-pre-processing" Oct 31 21:41:30.594537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594514 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2107fc-814c-4a1f-adfa-e2023e8c52bf" containerName="payload-pre-processing" Oct 31 21:41:30.594537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.594520 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="892582bc-6881-4e05-be5d-1fc72dc82847" containerName="payload-processing" Oct 31 21:41:30.599782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.599761 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.623516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.623483 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-c744c0-serving-cert\"" Oct 31 21:41:30.629984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.629948 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c"] Oct 31 21:41:30.719535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.719484 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.719740 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.719717 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d4f6-3591-451a-af67-933cc7cab8d4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.719789 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.719774 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.719867 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.719848 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd98r\" (UniqueName: \"kubernetes.io/projected/b652d4f6-3591-451a-af67-933cc7cab8d4-kube-api-access-nd98r\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.821053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.821012 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d4f6-3591-451a-af67-933cc7cab8d4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.821283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.821091 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.821283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.821132 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd98r\" (UniqueName: \"kubernetes.io/projected/b652d4f6-3591-451a-af67-933cc7cab8d4-kube-api-access-nd98r\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.821283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.821199 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.821949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.821919 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d4f6-3591-451a-af67-933cc7cab8d4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.823787 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.823761 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.823910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.823800 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.843120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.843087 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd98r\" (UniqueName: \"kubernetes.io/projected/b652d4f6-3591-451a-af67-933cc7cab8d4-kube-api-access-nd98r\") pod \"maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:30.916203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:30.916169 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:31.232566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:31.232533 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c"] Oct 31 21:41:31.233419 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:41:31.233386 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb652d4f6_3591_451a_af67_933cc7cab8d4.slice/crio-9d8ad6076fb5dcc5c6c08da86e72701fcec4e9ab7dd73fad0d9277256df02c1e WatchSource:0}: Error finding container 9d8ad6076fb5dcc5c6c08da86e72701fcec4e9ab7dd73fad0d9277256df02c1e: Status 404 returned error can't find the container with id 9d8ad6076fb5dcc5c6c08da86e72701fcec4e9ab7dd73fad0d9277256df02c1e Oct 31 21:41:32.220588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:32.220543 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" event={"ID":"b652d4f6-3591-451a-af67-933cc7cab8d4","Type":"ContainerStarted","Data":"4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728"} Oct 31 21:41:32.221118 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:32.220640 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:32.221118 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:32.220668 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" event={"ID":"b652d4f6-3591-451a-af67-933cc7cab8d4","Type":"ContainerStarted","Data":"9d8ad6076fb5dcc5c6c08da86e72701fcec4e9ab7dd73fad0d9277256df02c1e"} Oct 31 21:41:32.264475 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:32.264402 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" podStartSLOduration=1.856289346 podStartE2EDuration="2.26431708s" totalImagesPullingTime="408.027734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:41:32.259247468 +0000 UTC m=+1538.952204860" watchObservedRunningTime="2026-10-31 21:41:32.26431708 +0000 UTC m=+1538.957274474" Oct 31 21:41:32.539664 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:32.539579 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:41:36.176453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.176411 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957"] Oct 31 21:41:36.176985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.176710 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" podUID="917b637a-293e-4d77-8300-2369298b744f" containerName="maas-api" containerID="cri-o://8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4" gracePeriod=30 Oct 31 21:41:36.419865 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.419108 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6"] Oct 31 21:41:36.419865 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.419382 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" podUID="c9b209e1-d0bc-44fa-9e29-227b678d5e61" containerName="payload-pre-processing" containerID="cri-o://34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0" gracePeriod=30 Oct 31 21:41:36.568134 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.568095 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:41:36.725212 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.725114 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917b637a-293e-4d77-8300-2369298b744f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917b637a-293e-4d77-8300-2369298b744f-openshift-service-ca\") pod \"917b637a-293e-4d77-8300-2369298b744f\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " Oct 31 21:41:36.725409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.725216 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-metrics-certs\") pod \"917b637a-293e-4d77-8300-2369298b744f\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " Oct 31 21:41:36.725409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.725388 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-maas-api-tls\") pod \"917b637a-293e-4d77-8300-2369298b744f\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " Oct 31 21:41:36.725561 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.725489 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917b637a-293e-4d77-8300-2369298b744f-kube-api-access-nwgnw\" (UniqueName: \"kubernetes.io/projected/917b637a-293e-4d77-8300-2369298b744f-kube-api-access-nwgnw\") pod \"917b637a-293e-4d77-8300-2369298b744f\" (UID: \"917b637a-293e-4d77-8300-2369298b744f\") " Oct 31 21:41:36.727479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.727404 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917b637a-293e-4d77-8300-2369298b744f-openshift-service-ca" pod "917b637a-293e-4d77-8300-2369298b744f" (UID: "917b637a-293e-4d77-8300-2369298b744f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:36.732290 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.732249 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917b637a-293e-4d77-8300-2369298b744f-kube-api-access-nwgnw" pod "917b637a-293e-4d77-8300-2369298b744f" (UID: "917b637a-293e-4d77-8300-2369298b744f"). InnerVolumeSpecName "kube-api-access-nwgnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:36.732501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.732477 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-metrics-certs" pod "917b637a-293e-4d77-8300-2369298b744f" (UID: "917b637a-293e-4d77-8300-2369298b744f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:36.732864 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.732840 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-maas-api-tls" pod "917b637a-293e-4d77-8300-2369298b744f" (UID: "917b637a-293e-4d77-8300-2369298b744f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:36.828866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.827837 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/917b637a-293e-4d77-8300-2369298b744f-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:36.828866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.827879 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:36.828866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.827894 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/917b637a-293e-4d77-8300-2369298b744f-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:36.828866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.827907 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwgnw\" (UniqueName: \"kubernetes.io/projected/917b637a-293e-4d77-8300-2369298b744f-kube-api-access-nwgnw\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:36.854801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:36.854275 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:41:37.036565 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.036474 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b209e1-d0bc-44fa-9e29-227b678d5e61-kube-api-access-7rg48\" (UniqueName: \"kubernetes.io/projected/c9b209e1-d0bc-44fa-9e29-227b678d5e61-kube-api-access-7rg48\") pod \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\" (UID: \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\") " Oct 31 21:41:37.036774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.036662 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b209e1-d0bc-44fa-9e29-227b678d5e61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b209e1-d0bc-44fa-9e29-227b678d5e61-plugins-config-volume\") pod \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\" (UID: \"c9b209e1-d0bc-44fa-9e29-227b678d5e61\") " Oct 31 21:41:37.037088 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.037061 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b209e1-d0bc-44fa-9e29-227b678d5e61-plugins-config-volume" pod "c9b209e1-d0bc-44fa-9e29-227b678d5e61" (UID: "c9b209e1-d0bc-44fa-9e29-227b678d5e61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:37.038702 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.038677 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b209e1-d0bc-44fa-9e29-227b678d5e61-kube-api-access-7rg48" pod "c9b209e1-d0bc-44fa-9e29-227b678d5e61" (UID: "c9b209e1-d0bc-44fa-9e29-227b678d5e61"). InnerVolumeSpecName "kube-api-access-7rg48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:37.138556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.138489 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b209e1-d0bc-44fa-9e29-227b678d5e61-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:37.138748 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.138563 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rg48\" (UniqueName: \"kubernetes.io/projected/c9b209e1-d0bc-44fa-9e29-227b678d5e61-kube-api-access-7rg48\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:37.265559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.265516 5188 generic.go:356] "Generic (PLEG): container finished" podID="917b637a-293e-4d77-8300-2369298b744f" containerID="8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4" exitCode=0 Oct 31 21:41:37.266044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.265606 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" Oct 31 21:41:37.266044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.265603 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" event={"ID":"917b637a-293e-4d77-8300-2369298b744f","Type":"ContainerDied","Data":"8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4"} Oct 31 21:41:37.266044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.265763 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957" event={"ID":"917b637a-293e-4d77-8300-2369298b744f","Type":"ContainerDied","Data":"d4e2017e686bfb14133b7876cb8d5e0aa53e223fe90c5b7faf103a402dfd303a"} Oct 31 21:41:37.266044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.265793 5188 scope.go:122] "RemoveContainer" containerID="8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4" Oct 31 21:41:37.267762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.267733 5188 generic.go:356] "Generic (PLEG): container finished" podID="c9b209e1-d0bc-44fa-9e29-227b678d5e61" containerID="34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0" exitCode=0 Oct 31 21:41:37.267888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.267789 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" Oct 31 21:41:37.267888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.267788 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" event={"ID":"c9b209e1-d0bc-44fa-9e29-227b678d5e61","Type":"ContainerDied","Data":"34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0"} Oct 31 21:41:37.268022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.267911 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6" event={"ID":"c9b209e1-d0bc-44fa-9e29-227b678d5e61","Type":"ContainerDied","Data":"d175c4f1b87fb6ef723e8e9f7a279d4bba7e0684410f46a2d22e794da77cd4ae"} Oct 31 21:41:37.280173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.280075 5188 scope.go:122] "RemoveContainer" containerID="8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4" Oct 31 21:41:37.280656 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:37.280624 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4\": container with ID starting with 8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4 not found: ID does not exist" containerID="8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4" Oct 31 21:41:37.280762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.280667 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4"} err="failed to get container status \"8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4\": rpc error: code = NotFound desc = could not find container \"8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4\": container with ID starting with 8406285b3656c5a7020ae273e5238758c2c1d3af57b51a6157c534398e4356b4 not found: ID does not exist" Oct 31 21:41:37.280762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.280685 5188 scope.go:122] "RemoveContainer" containerID="34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0" Oct 31 21:41:37.295463 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.295338 5188 scope.go:122] "RemoveContainer" containerID="34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0" Oct 31 21:41:37.295776 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:37.295750 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0\": container with ID starting with 34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0 not found: ID does not exist" containerID="34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0" Oct 31 21:41:37.295892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.295781 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0"} err="failed to get container status \"34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0\": rpc error: code = NotFound desc = could not find container \"34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0\": container with ID starting with 34edc901c510f2c82204830f8f1784235dffa7f3f3b48f3e32673f63832616c0 not found: ID does not exist" Oct 31 21:41:37.329230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.329194 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957"] Oct 31 21:41:37.333339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.333292 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-752623-6578479477-7c957"] Oct 31 21:41:37.361136 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.361104 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6"] Oct 31 21:41:37.367127 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.367081 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-752623-b4dcbd679-27pb6"] Oct 31 21:41:37.474761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.474726 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917b637a-293e-4d77-8300-2369298b744f" path="/var/lib/kubelet/pods/917b637a-293e-4d77-8300-2369298b744f/volumes" Oct 31 21:41:37.475246 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:37.475230 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b209e1-d0bc-44fa-9e29-227b678d5e61" path="/var/lib/kubelet/pods/c9b209e1-d0bc-44fa-9e29-227b678d5e61/volumes" Oct 31 21:41:38.236905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:38.236872 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:41:41.032953 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:41.032907 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp"] Oct 31 21:41:41.042617 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:41.042583 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8424d9d-cjrdp"] Oct 31 21:41:41.474673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:41.474626 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ebdccf2-7fce-4054-a600-e50208152f4d" path="/var/lib/kubelet/pods/8ebdccf2-7fce-4054-a600-e50208152f4d/volumes" Oct 31 21:41:44.730362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:44.728966 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c"] Oct 31 21:41:44.730362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:44.729259 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" containerID="cri-o://90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89" gracePeriod=30 Oct 31 21:41:50.008481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.008407 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:41:50.149255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149220 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-podinfo\") pod \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " Oct 31 21:41:50.149451 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149316 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-socket\") pod \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " Oct 31 21:41:50.149451 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149411 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-credential-socket\") pod \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " Oct 31 21:41:50.149594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149459 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-envoy\") pod \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " Oct 31 21:41:50.149594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149511 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istiod-ca-cert\") pod \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " Oct 31 21:41:50.149594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149551 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-data\") pod \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " Oct 31 21:41:50.149751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149649 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-socket" pod "6255c981-759d-4a87-b6a1-dfb15b19fd6c" (UID: "6255c981-759d-4a87-b6a1-dfb15b19fd6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:50.149751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149681 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-kube-api-access-kcmpd\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-kube-api-access-kcmpd\") pod \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " Oct 31 21:41:50.149751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149735 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-credential-socket" pod "6255c981-759d-4a87-b6a1-dfb15b19fd6c" (UID: "6255c981-759d-4a87-b6a1-dfb15b19fd6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:50.149906 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149747 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-token\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-token\") pod \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " Oct 31 21:41:50.149906 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149856 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-certs\") pod \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\" (UID: \"6255c981-759d-4a87-b6a1-dfb15b19fd6c\") " Oct 31 21:41:50.149906 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149888 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-data" pod "6255c981-759d-4a87-b6a1-dfb15b19fd6c" (UID: "6255c981-759d-4a87-b6a1-dfb15b19fd6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:50.150124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.149975 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istiod-ca-cert" pod "6255c981-759d-4a87-b6a1-dfb15b19fd6c" (UID: "6255c981-759d-4a87-b6a1-dfb15b19fd6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:50.150396 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.150367 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-certs" pod "6255c981-759d-4a87-b6a1-dfb15b19fd6c" (UID: "6255c981-759d-4a87-b6a1-dfb15b19fd6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:50.150479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.150460 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.150543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.150486 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.150543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.150500 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.150543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.150513 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.152128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.152093 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-envoy" pod "6255c981-759d-4a87-b6a1-dfb15b19fd6c" (UID: "6255c981-759d-4a87-b6a1-dfb15b19fd6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:50.152273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.152254 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-kube-api-access-kcmpd" pod "6255c981-759d-4a87-b6a1-dfb15b19fd6c" (UID: "6255c981-759d-4a87-b6a1-dfb15b19fd6c"). InnerVolumeSpecName "kube-api-access-kcmpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:50.152363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.152311 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-podinfo" pod "6255c981-759d-4a87-b6a1-dfb15b19fd6c" (UID: "6255c981-759d-4a87-b6a1-dfb15b19fd6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:41:50.152492 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.152476 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-token" pod "6255c981-759d-4a87-b6a1-dfb15b19fd6c" (UID: "6255c981-759d-4a87-b6a1-dfb15b19fd6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:50.247342 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.247278 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb"] Oct 31 21:41:50.248004 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.247946 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" podUID="c57e1750-8e39-4c03-aab6-c58dcb835b0a" containerName="maas-api" containerID="cri-o://ff1ecdd7c8c1378b05d10172abfdb0a27e49664c3c5e128243cfc5db442c0d0a" gracePeriod=30 Oct 31 21:41:50.253366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.253339 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcmpd\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-kube-api-access-kcmpd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.253564 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.253549 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.254105 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.254079 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.254105 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.254105 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.254273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.254116 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6255c981-759d-4a87-b6a1-dfb15b19fd6c-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.389791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.389737 5188 generic.go:356] "Generic (PLEG): container finished" podID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerID="90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89" exitCode=0 Oct 31 21:41:50.389985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.389822 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" Oct 31 21:41:50.389985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.389818 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" event={"ID":"6255c981-759d-4a87-b6a1-dfb15b19fd6c","Type":"ContainerDied","Data":"90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89"} Oct 31 21:41:50.389985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.389967 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c" event={"ID":"6255c981-759d-4a87-b6a1-dfb15b19fd6c","Type":"ContainerDied","Data":"cd9f6e72446081933dbc742ce6f5a927c0aec254cd14c1c6dae42b7895a6c86d"} Oct 31 21:41:50.390147 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.390009 5188 scope.go:122] "RemoveContainer" containerID="90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89" Oct 31 21:41:50.395743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.395719 5188 generic.go:356] "Generic (PLEG): container finished" podID="c57e1750-8e39-4c03-aab6-c58dcb835b0a" containerID="ff1ecdd7c8c1378b05d10172abfdb0a27e49664c3c5e128243cfc5db442c0d0a" exitCode=0 Oct 31 21:41:50.395908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.395883 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" event={"ID":"c57e1750-8e39-4c03-aab6-c58dcb835b0a","Type":"ContainerDied","Data":"ff1ecdd7c8c1378b05d10172abfdb0a27e49664c3c5e128243cfc5db442c0d0a"} Oct 31 21:41:50.409898 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.409875 5188 scope.go:122] "RemoveContainer" containerID="e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe" Oct 31 21:41:50.421909 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.421881 5188 scope.go:122] "RemoveContainer" containerID="90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89" Oct 31 21:41:50.422353 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:50.422212 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89\": container with ID starting with 90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89 not found: ID does not exist" containerID="90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89" Oct 31 21:41:50.422353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.422244 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89"} err="failed to get container status \"90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89\": rpc error: code = NotFound desc = could not find container \"90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89\": container with ID starting with 90c2b7962460891d820e470f20323c79515c6b522b345146d66cbee83028ce89 not found: ID does not exist" Oct 31 21:41:50.422353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.422263 5188 scope.go:122] "RemoveContainer" containerID="e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe" Oct 31 21:41:50.422745 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:41:50.422721 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe\": container with ID starting with e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe not found: ID does not exist" containerID="e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe" Oct 31 21:41:50.422824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.422752 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe"} err="failed to get container status \"e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe\": rpc error: code = NotFound desc = could not find container \"e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe\": container with ID starting with e70123aca571679b8b8d2dbc43f75762d1a5ff9a2f1060f7d47dee47cbfd92fe not found: ID does not exist" Oct 31 21:41:50.443277 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.443256 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c"] Oct 31 21:41:50.443463 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.443448 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-752623-openshift-default-f95fb57f9-njt2c"] Oct 31 21:41:50.553616 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.553580 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:41:50.660984 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.660946 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57e1750-8e39-4c03-aab6-c58dcb835b0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57e1750-8e39-4c03-aab6-c58dcb835b0a-openshift-service-ca\") pod \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " Oct 31 21:41:50.661191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.661050 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-metrics-certs\") pod \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " Oct 31 21:41:50.661191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.661154 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-maas-api-tls\") pod \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " Oct 31 21:41:50.661386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.661212 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57e1750-8e39-4c03-aab6-c58dcb835b0a-kube-api-access-ch88v\" (UniqueName: \"kubernetes.io/projected/c57e1750-8e39-4c03-aab6-c58dcb835b0a-kube-api-access-ch88v\") pod \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\" (UID: \"c57e1750-8e39-4c03-aab6-c58dcb835b0a\") " Oct 31 21:41:50.664693 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.664654 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-metrics-certs" pod "c57e1750-8e39-4c03-aab6-c58dcb835b0a" (UID: "c57e1750-8e39-4c03-aab6-c58dcb835b0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:50.664951 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.664639 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57e1750-8e39-4c03-aab6-c58dcb835b0a-kube-api-access-ch88v" pod "c57e1750-8e39-4c03-aab6-c58dcb835b0a" (UID: "c57e1750-8e39-4c03-aab6-c58dcb835b0a"). InnerVolumeSpecName "kube-api-access-ch88v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:50.665370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.665308 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57e1750-8e39-4c03-aab6-c58dcb835b0a-openshift-service-ca" pod "c57e1750-8e39-4c03-aab6-c58dcb835b0a" (UID: "c57e1750-8e39-4c03-aab6-c58dcb835b0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:50.668999 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.668971 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-maas-api-tls" pod "c57e1750-8e39-4c03-aab6-c58dcb835b0a" (UID: "c57e1750-8e39-4c03-aab6-c58dcb835b0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:50.770383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.770344 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57e1750-8e39-4c03-aab6-c58dcb835b0a-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.770580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.770386 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.770580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.770403 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c57e1750-8e39-4c03-aab6-c58dcb835b0a-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:50.770580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:50.770416 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch88v\" (UniqueName: \"kubernetes.io/projected/c57e1750-8e39-4c03-aab6-c58dcb835b0a-kube-api-access-ch88v\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:41:51.135767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.135728 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l"] Oct 31 21:41:51.140905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.140875 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3dae1-wn47l"] Oct 31 21:41:51.408667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.408571 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" Oct 31 21:41:51.408667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.408596 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb" event={"ID":"c57e1750-8e39-4c03-aab6-c58dcb835b0a","Type":"ContainerDied","Data":"86ccc13214ca640ff854f5dcb51cd4cdfaa088b7903a0fe90811e6d9a289c4ae"} Oct 31 21:41:51.408667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.408665 5188 scope.go:122] "RemoveContainer" containerID="ff1ecdd7c8c1378b05d10172abfdb0a27e49664c3c5e128243cfc5db442c0d0a" Oct 31 21:41:51.454481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.454448 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb"] Oct 31 21:41:51.460871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.460840 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3dae1-75f5bc7db6-dp2gb"] Oct 31 21:41:51.473883 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.473849 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" path="/var/lib/kubelet/pods/6255c981-759d-4a87-b6a1-dfb15b19fd6c/volumes" Oct 31 21:41:51.474354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.474338 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57e1750-8e39-4c03-aab6-c58dcb835b0a" path="/var/lib/kubelet/pods/c57e1750-8e39-4c03-aab6-c58dcb835b0a/volumes" Oct 31 21:41:51.474786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:51.474772 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45bf814-a94a-4bdf-bbf4-00591f7538a4" path="/var/lib/kubelet/pods/f45bf814-a94a-4bdf-bbf4-00591f7538a4/volumes" Oct 31 21:41:52.413356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.413308 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq"] Oct 31 21:41:52.413920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.413904 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c9b209e1-d0bc-44fa-9e29-227b678d5e61" containerName="payload-pre-processing" Oct 31 21:41:52.413920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.413919 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b209e1-d0bc-44fa-9e29-227b678d5e61" containerName="payload-pre-processing" Oct 31 21:41:52.414035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.413938 5188 cpu_manager.go:414] "containerMap: removing container" podUID="917b637a-293e-4d77-8300-2369298b744f" containerName="maas-api" Oct 31 21:41:52.414035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.413944 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917b637a-293e-4d77-8300-2369298b744f" containerName="maas-api" Oct 31 21:41:52.414035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.413953 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c57e1750-8e39-4c03-aab6-c58dcb835b0a" containerName="maas-api" Oct 31 21:41:52.414035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.413958 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57e1750-8e39-4c03-aab6-c58dcb835b0a" containerName="maas-api" Oct 31 21:41:52.414035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.413984 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" Oct 31 21:41:52.414035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.413989 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" Oct 31 21:41:52.414035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.414004 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" Oct 31 21:41:52.414035 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.414009 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" Oct 31 21:41:52.414439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.414090 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b209e1-d0bc-44fa-9e29-227b678d5e61" containerName="payload-pre-processing" Oct 31 21:41:52.414439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.414102 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="917b637a-293e-4d77-8300-2369298b744f" containerName="maas-api" Oct 31 21:41:52.414439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.414108 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57e1750-8e39-4c03-aab6-c58dcb835b0a" containerName="maas-api" Oct 31 21:41:52.414439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.414118 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" Oct 31 21:41:52.419631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.419586 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:41:52.436211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.436178 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq"] Oct 31 21:41:52.441673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.441639 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6d6576\"" Oct 31 21:41:52.441673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.441670 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6d6576-dockercfg-r2kjf\"" Oct 31 21:41:52.533349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.528635 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52"] Oct 31 21:41:52.533349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.529965 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6255c981-759d-4a87-b6a1-dfb15b19fd6c" containerName="istio-proxy" Oct 31 21:41:52.550347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.543131 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:41:52.550347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.548987 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52"] Oct 31 21:41:52.596299 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.596261 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-225gk\" (UniqueName: \"kubernetes.io/projected/962b779d-da15-4513-ade5-b5a289b411dd-kube-api-access-225gk\") pod \"payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq\" (UID: \"962b779d-da15-4513-ade5-b5a289b411dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:41:52.596562 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.596447 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962b779d-da15-4513-ade5-b5a289b411dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq\" (UID: \"962b779d-da15-4513-ade5-b5a289b411dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:41:52.698670 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.698566 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzxx5\" (UniqueName: \"kubernetes.io/projected/82289a34-8b1f-42cf-b716-f21732bcf54f-kube-api-access-lzxx5\") pod \"payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52\" (UID: \"82289a34-8b1f-42cf-b716-f21732bcf54f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:41:52.698861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.698829 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-225gk\" (UniqueName: \"kubernetes.io/projected/962b779d-da15-4513-ade5-b5a289b411dd-kube-api-access-225gk\") pod \"payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq\" (UID: \"962b779d-da15-4513-ade5-b5a289b411dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:41:52.698928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.698870 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962b779d-da15-4513-ade5-b5a289b411dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq\" (UID: \"962b779d-da15-4513-ade5-b5a289b411dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:41:52.698928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.698910 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82289a34-8b1f-42cf-b716-f21732bcf54f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52\" (UID: \"82289a34-8b1f-42cf-b716-f21732bcf54f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:41:52.699661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.699638 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962b779d-da15-4513-ade5-b5a289b411dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq\" (UID: \"962b779d-da15-4513-ade5-b5a289b411dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:41:52.720705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.720670 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-225gk\" (UniqueName: \"kubernetes.io/projected/962b779d-da15-4513-ade5-b5a289b411dd-kube-api-access-225gk\") pod \"payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq\" (UID: \"962b779d-da15-4513-ade5-b5a289b411dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:41:52.735379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.735351 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:41:52.800072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.800034 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82289a34-8b1f-42cf-b716-f21732bcf54f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52\" (UID: \"82289a34-8b1f-42cf-b716-f21732bcf54f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:41:52.800954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.800917 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82289a34-8b1f-42cf-b716-f21732bcf54f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52\" (UID: \"82289a34-8b1f-42cf-b716-f21732bcf54f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:41:52.801086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.801059 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzxx5\" (UniqueName: \"kubernetes.io/projected/82289a34-8b1f-42cf-b716-f21732bcf54f-kube-api-access-lzxx5\") pod \"payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52\" (UID: \"82289a34-8b1f-42cf-b716-f21732bcf54f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:41:52.824383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.822143 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b"] Oct 31 21:41:52.825139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.825112 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzxx5\" (UniqueName: \"kubernetes.io/projected/82289a34-8b1f-42cf-b716-f21732bcf54f-kube-api-access-lzxx5\") pod \"payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52\" (UID: \"82289a34-8b1f-42cf-b716-f21732bcf54f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:41:52.834668 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.833730 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:52.848745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.847741 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b"] Oct 31 21:41:52.902431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.902387 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11611511-4c92-4cd5-81a1-c15b30598446-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:52.902626 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.902490 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:52.902626 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.902580 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56hg6\" (UniqueName: \"kubernetes.io/projected/11611511-4c92-4cd5-81a1-c15b30598446-kube-api-access-56hg6\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:52.902749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.902692 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:52.906129 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:52.906103 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:41:53.025587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.022455 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:53.025587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.022591 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11611511-4c92-4cd5-81a1-c15b30598446-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:53.025587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.022686 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:53.025587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.022776 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56hg6\" (UniqueName: \"kubernetes.io/projected/11611511-4c92-4cd5-81a1-c15b30598446-kube-api-access-56hg6\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:53.025587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.024879 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11611511-4c92-4cd5-81a1-c15b30598446-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:53.035808 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.034933 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:53.037355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.037244 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:53.045503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.045223 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56hg6\" (UniqueName: \"kubernetes.io/projected/11611511-4c92-4cd5-81a1-c15b30598446-kube-api-access-56hg6\") pod \"maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:53.067578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.067520 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq"] Oct 31 21:41:53.072706 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:41:53.072648 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962b779d_da15_4513_ade5_b5a289b411dd.slice/crio-b28ba4e020ecf8dc0448731c141b2790f7bc2a9c3651c6cb8d8473bfc08421ee WatchSource:0}: Error finding container b28ba4e020ecf8dc0448731c141b2790f7bc2a9c3651c6cb8d8473bfc08421ee: Status 404 returned error can't find the container with id b28ba4e020ecf8dc0448731c141b2790f7bc2a9c3651c6cb8d8473bfc08421ee Oct 31 21:41:53.167496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.167470 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:53.238594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.238566 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52"] Oct 31 21:41:53.431342 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.431288 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" event={"ID":"962b779d-da15-4513-ade5-b5a289b411dd","Type":"ContainerStarted","Data":"bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c"} Oct 31 21:41:53.431844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.431399 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" event={"ID":"962b779d-da15-4513-ade5-b5a289b411dd","Type":"ContainerStarted","Data":"b28ba4e020ecf8dc0448731c141b2790f7bc2a9c3651c6cb8d8473bfc08421ee"} Oct 31 21:41:53.431844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.431513 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:41:53.433183 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.433160 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" event={"ID":"82289a34-8b1f-42cf-b716-f21732bcf54f","Type":"ContainerStarted","Data":"bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9"} Oct 31 21:41:53.433288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.433200 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:41:53.433288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.433216 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" event={"ID":"82289a34-8b1f-42cf-b716-f21732bcf54f","Type":"ContainerStarted","Data":"405610b5de84b590e5e2d5e49bbdb96149a27fbfcd4f1376e1b73c8e11878b95"} Oct 31 21:41:53.472097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.472044 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" podStartSLOduration=1.472028524 podStartE2EDuration="1.472028524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:41:53.466939941 +0000 UTC m=+1560.159897332" watchObservedRunningTime="2026-10-31 21:41:53.472028524 +0000 UTC m=+1560.164985952" Oct 31 21:41:53.482977 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.482943 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b"] Oct 31 21:41:53.483119 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:41:53.483077 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11611511_4c92_4cd5_81a1_c15b30598446.slice/crio-11b26386ceb5a73e2f02ede0d41a8ef881ea1d3adbe0c816cdf5d1f93aa41708 WatchSource:0}: Error finding container 11b26386ceb5a73e2f02ede0d41a8ef881ea1d3adbe0c816cdf5d1f93aa41708: Status 404 returned error can't find the container with id 11b26386ceb5a73e2f02ede0d41a8ef881ea1d3adbe0c816cdf5d1f93aa41708 Oct 31 21:41:53.514599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:53.514536 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" podStartSLOduration=1.514517299 podStartE2EDuration="1.514517299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:41:53.510190399 +0000 UTC m=+1560.203147792" watchObservedRunningTime="2026-10-31 21:41:53.514517299 +0000 UTC m=+1560.207474690" Oct 31 21:41:54.443379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:54.443336 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" event={"ID":"11611511-4c92-4cd5-81a1-c15b30598446","Type":"ContainerStarted","Data":"50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730"} Oct 31 21:41:54.443891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:54.443403 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" event={"ID":"11611511-4c92-4cd5-81a1-c15b30598446","Type":"ContainerStarted","Data":"11b26386ceb5a73e2f02ede0d41a8ef881ea1d3adbe0c816cdf5d1f93aa41708"} Oct 31 21:41:54.444087 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:54.444057 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:41:54.482623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:54.482565 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" podStartSLOduration=2.197752726 podStartE2EDuration="2.482543632s" totalImagesPullingTime="284.790906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:41:54.480718077 +0000 UTC m=+1561.173675470" watchObservedRunningTime="2026-10-31 21:41:54.482543632 +0000 UTC m=+1561.175501047" Oct 31 21:41:55.679414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:55.679372 5188 scope.go:122] "RemoveContainer" containerID="dc2f92a0142bfbfeb15861cdbacacebebe31ec5db5974716893109a961d03454" Oct 31 21:41:55.691179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:55.691147 5188 scope.go:122] "RemoveContainer" containerID="a71e48cfb0fea0588fbc4e3ea6815e1d70aa01ead7b915dc2bbef179ae5ce263" Oct 31 21:41:55.702885 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:55.702862 5188 scope.go:122] "RemoveContainer" containerID="12558d84e09fd46c05ef80f5187a8c8472ee605145854a93cf7f2648ecc2d85d" Oct 31 21:41:55.714046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:55.714018 5188 scope.go:122] "RemoveContainer" containerID="09ff79b82f8af61395715aa7c7951f17e5eb98b864730cf38e1942a5d2c68a88" Oct 31 21:41:56.092054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:56.091955 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt_d2104e85-d811-4b12-9628-226615b41f16/payload-processing/0.log" Oct 31 21:41:56.694024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:56.693987 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8_7f20f387-c81d-41eb-804b-6b992484f56b/payload-processing/0.log" Oct 31 21:41:56.877956 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:56.877917 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8_7f20f387-c81d-41eb-804b-6b992484f56b/payload-processing/0.log" Oct 31 21:41:58.060103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:58.060064 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q"] Oct 31 21:41:58.060694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:41:58.060369 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" containerID="cri-o://07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120" gracePeriod=30 Oct 31 21:42:01.134037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.133987 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q"] Oct 31 21:42:01.473835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.473756 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 21:42:01.550130 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.550081 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c"] Oct 31 21:42:01.550465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.550405 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" podUID="b652d4f6-3591-451a-af67-933cc7cab8d4" containerName="maas-api" containerID="cri-o://4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728" gracePeriod=30 Oct 31 21:42:01.856223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.856193 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:42:01.858177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.858156 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b652d4f6-3591-451a-af67-933cc7cab8d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d4f6-3591-451a-af67-933cc7cab8d4-openshift-service-ca\") pod \"b652d4f6-3591-451a-af67-933cc7cab8d4\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " Oct 31 21:42:01.858440 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.858419 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-metrics-certs\") pod \"b652d4f6-3591-451a-af67-933cc7cab8d4\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " Oct 31 21:42:01.858487 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.858454 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b652d4f6-3591-451a-af67-933cc7cab8d4-kube-api-access-nd98r\" (UniqueName: \"kubernetes.io/projected/b652d4f6-3591-451a-af67-933cc7cab8d4-kube-api-access-nd98r\") pod \"b652d4f6-3591-451a-af67-933cc7cab8d4\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " Oct 31 21:42:01.858535 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.858524 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-maas-api-tls\") pod \"b652d4f6-3591-451a-af67-933cc7cab8d4\" (UID: \"b652d4f6-3591-451a-af67-933cc7cab8d4\") " Oct 31 21:42:01.858588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.858563 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b652d4f6-3591-451a-af67-933cc7cab8d4-openshift-service-ca" pod "b652d4f6-3591-451a-af67-933cc7cab8d4" (UID: "b652d4f6-3591-451a-af67-933cc7cab8d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:01.859052 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.859025 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b652d4f6-3591-451a-af67-933cc7cab8d4-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:01.860716 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.860674 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-metrics-certs" pod "b652d4f6-3591-451a-af67-933cc7cab8d4" (UID: "b652d4f6-3591-451a-af67-933cc7cab8d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:42:01.860846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.860819 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b652d4f6-3591-451a-af67-933cc7cab8d4-kube-api-access-nd98r" pod "b652d4f6-3591-451a-af67-933cc7cab8d4" (UID: "b652d4f6-3591-451a-af67-933cc7cab8d4"). InnerVolumeSpecName "kube-api-access-nd98r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:01.860983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.860968 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-maas-api-tls" pod "b652d4f6-3591-451a-af67-933cc7cab8d4" (UID: "b652d4f6-3591-451a-af67-933cc7cab8d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:42:01.960125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.960089 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:01.960125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.960123 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b652d4f6-3591-451a-af67-933cc7cab8d4-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:01.960405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:01.960143 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd98r\" (UniqueName: \"kubernetes.io/projected/b652d4f6-3591-451a-af67-933cc7cab8d4-kube-api-access-nd98r\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:02.517244 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:02.517206 5188 generic.go:356] "Generic (PLEG): container finished" podID="b652d4f6-3591-451a-af67-933cc7cab8d4" containerID="4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728" exitCode=0 Oct 31 21:42:02.517775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:02.517267 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" Oct 31 21:42:02.517775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:02.517375 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" event={"ID":"b652d4f6-3591-451a-af67-933cc7cab8d4","Type":"ContainerDied","Data":"4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728"} Oct 31 21:42:02.517775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:02.517416 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c" event={"ID":"b652d4f6-3591-451a-af67-933cc7cab8d4","Type":"ContainerDied","Data":"9d8ad6076fb5dcc5c6c08da86e72701fcec4e9ab7dd73fad0d9277256df02c1e"} Oct 31 21:42:02.517775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:02.517441 5188 scope.go:122] "RemoveContainer" containerID="4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728" Oct 31 21:42:02.537055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:02.536449 5188 scope.go:122] "RemoveContainer" containerID="4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728" Oct 31 21:42:02.537055 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:02.536852 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728\": container with ID starting with 4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728 not found: ID does not exist" containerID="4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728" Oct 31 21:42:02.537055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:02.536883 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728"} err="failed to get container status \"4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728\": rpc error: code = NotFound desc = could not find container \"4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728\": container with ID starting with 4f0588ae12ab2903b44849af61facf83e19c67f9cbfb1a3048f1e69b698e2728 not found: ID does not exist" Oct 31 21:42:02.551747 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:02.551711 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:42:02.570197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:02.570160 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c"] Oct 31 21:42:02.580083 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:02.580050 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-7dfcbc7bbc-6r76c"] Oct 31 21:42:03.376742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.376713 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:42:03.480838 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.480787 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b652d4f6-3591-451a-af67-933cc7cab8d4" path="/var/lib/kubelet/pods/b652d4f6-3591-451a-af67-933cc7cab8d4/volumes" Oct 31 21:42:03.486673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.486641 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-istio-token\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-istio-token\") pod \"b0a9a747-3941-47bf-8765-fe70dd146648\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " Oct 31 21:42:03.486853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.486725 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-envoy\") pod \"b0a9a747-3941-47bf-8765-fe70dd146648\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " Oct 31 21:42:03.486853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.486830 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-socket\") pod \"b0a9a747-3941-47bf-8765-fe70dd146648\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " Oct 31 21:42:03.486965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.486896 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-data\") pod \"b0a9a747-3941-47bf-8765-fe70dd146648\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " Oct 31 21:42:03.486965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.486951 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a9a747-3941-47bf-8765-fe70dd146648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a9a747-3941-47bf-8765-fe70dd146648-istiod-ca-cert\") pod \"b0a9a747-3941-47bf-8765-fe70dd146648\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " Oct 31 21:42:03.487072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.487015 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0a9a747-3941-47bf-8765-fe70dd146648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a9a747-3941-47bf-8765-fe70dd146648-istio-podinfo\") pod \"b0a9a747-3941-47bf-8765-fe70dd146648\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " Oct 31 21:42:03.487072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.487051 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-kube-api-access-cmkrw\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-kube-api-access-cmkrw\") pod \"b0a9a747-3941-47bf-8765-fe70dd146648\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " Oct 31 21:42:03.487177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.487129 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-credential-socket\") pod \"b0a9a747-3941-47bf-8765-fe70dd146648\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " Oct 31 21:42:03.487229 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.487182 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-certs\") pod \"b0a9a747-3941-47bf-8765-fe70dd146648\" (UID: \"b0a9a747-3941-47bf-8765-fe70dd146648\") " Oct 31 21:42:03.487549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.487523 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-socket" pod "b0a9a747-3941-47bf-8765-fe70dd146648" (UID: "b0a9a747-3941-47bf-8765-fe70dd146648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:03.487749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.487655 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-data" pod "b0a9a747-3941-47bf-8765-fe70dd146648" (UID: "b0a9a747-3941-47bf-8765-fe70dd146648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:03.487963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.487943 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:03.488079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.488062 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:03.488079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.487968 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-certs" pod "b0a9a747-3941-47bf-8765-fe70dd146648" (UID: "b0a9a747-3941-47bf-8765-fe70dd146648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:03.488214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.488015 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-credential-socket" pod "b0a9a747-3941-47bf-8765-fe70dd146648" (UID: "b0a9a747-3941-47bf-8765-fe70dd146648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:03.488214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.488030 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a9a747-3941-47bf-8765-fe70dd146648-istiod-ca-cert" pod "b0a9a747-3941-47bf-8765-fe70dd146648" (UID: "b0a9a747-3941-47bf-8765-fe70dd146648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:03.490262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.490182 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-kube-api-access-cmkrw" pod "b0a9a747-3941-47bf-8765-fe70dd146648" (UID: "b0a9a747-3941-47bf-8765-fe70dd146648"). InnerVolumeSpecName "kube-api-access-cmkrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:03.490559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.490517 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0a9a747-3941-47bf-8765-fe70dd146648-istio-podinfo" pod "b0a9a747-3941-47bf-8765-fe70dd146648" (UID: "b0a9a747-3941-47bf-8765-fe70dd146648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:42:03.490559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.490535 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-istio-token" pod "b0a9a747-3941-47bf-8765-fe70dd146648" (UID: "b0a9a747-3941-47bf-8765-fe70dd146648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:03.490703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.490670 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-envoy" pod "b0a9a747-3941-47bf-8765-fe70dd146648" (UID: "b0a9a747-3941-47bf-8765-fe70dd146648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:03.527915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.527873 5188 generic.go:356] "Generic (PLEG): container finished" podID="b0a9a747-3941-47bf-8765-fe70dd146648" containerID="07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120" exitCode=0 Oct 31 21:42:03.528469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.527948 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" Oct 31 21:42:03.528469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.527954 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" event={"ID":"b0a9a747-3941-47bf-8765-fe70dd146648","Type":"ContainerDied","Data":"07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120"} Oct 31 21:42:03.528469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.528041 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q" event={"ID":"b0a9a747-3941-47bf-8765-fe70dd146648","Type":"ContainerDied","Data":"6ae22eba0bc0f6b3f38ce973bbc1d4ea67c597aaa59f9f33c0fbaed250a3a1fe"} Oct 31 21:42:03.528469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.528068 5188 scope.go:122] "RemoveContainer" containerID="07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120" Oct 31 21:42:03.544191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.544168 5188 scope.go:122] "RemoveContainer" containerID="929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6" Oct 31 21:42:03.554970 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.554946 5188 scope.go:122] "RemoveContainer" containerID="07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120" Oct 31 21:42:03.555286 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:03.555264 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120\": container with ID starting with 07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120 not found: ID does not exist" containerID="07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120" Oct 31 21:42:03.555390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.555297 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120"} err="failed to get container status \"07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120\": rpc error: code = NotFound desc = could not find container \"07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120\": container with ID starting with 07d9fa6e1d94023999aa04795503c2156253fd5d65cf63166160e29931931120 not found: ID does not exist" Oct 31 21:42:03.555390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.555317 5188 scope.go:122] "RemoveContainer" containerID="929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6" Oct 31 21:42:03.555682 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:03.555663 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6\": container with ID starting with 929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6 not found: ID does not exist" containerID="929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6" Oct 31 21:42:03.555735 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.555687 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6"} err="failed to get container status \"929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6\": rpc error: code = NotFound desc = could not find container \"929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6\": container with ID starting with 929b5615f77813ebde30c47d471c9f540721c92a8a2cbd7eea85c5cc953809d6 not found: ID does not exist" Oct 31 21:42:03.574369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.574335 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q"] Oct 31 21:42:03.582241 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.582213 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a3dae1-openshift-default-54785768d5-q8c9q"] Oct 31 21:42:03.589042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.589013 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a9a747-3941-47bf-8765-fe70dd146648-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:03.589042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.589040 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkrw\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-kube-api-access-cmkrw\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:03.589252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.589051 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:03.589252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.589062 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:03.589252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.589071 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a9a747-3941-47bf-8765-fe70dd146648-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:03.589252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.589080 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a9a747-3941-47bf-8765-fe70dd146648-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:03.589252 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:03.589092 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a9a747-3941-47bf-8765-fe70dd146648-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:04.447583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:04.447496 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:42:04.447583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:04.447580 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:42:05.474641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.474602 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" path="/var/lib/kubelet/pods/b0a9a747-3941-47bf-8765-fe70dd146648/volumes" Oct 31 21:42:05.489078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.489041 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4"] Oct 31 21:42:05.490027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.490006 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b652d4f6-3591-451a-af67-933cc7cab8d4" containerName="maas-api" Oct 31 21:42:05.490133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.490031 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b652d4f6-3591-451a-af67-933cc7cab8d4" containerName="maas-api" Oct 31 21:42:05.490133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.490072 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" Oct 31 21:42:05.490133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.490081 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" Oct 31 21:42:05.490133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.490119 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" Oct 31 21:42:05.490133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.490127 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" Oct 31 21:42:05.490290 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.490255 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b652d4f6-3591-451a-af67-933cc7cab8d4" containerName="maas-api" Oct 31 21:42:05.490290 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.490275 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" Oct 31 21:42:05.496196 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.496170 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:05.515117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.515080 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4"] Oct 31 21:42:05.517798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.517704 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d43b66-dockercfg-8qck6\"" Oct 31 21:42:05.517798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.517791 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d43b66\"" Oct 31 21:42:05.604347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.603768 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8"] Oct 31 21:42:05.607416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.604913 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a9a747-3941-47bf-8765-fe70dd146648" containerName="istio-proxy" Oct 31 21:42:05.609476 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.609450 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:05.619507 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.619474 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxzph\" (UniqueName: \"kubernetes.io/projected/8ac7a720-aaab-4619-b3be-a0799a463093-kube-api-access-pxzph\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4\" (UID: \"8ac7a720-aaab-4619-b3be-a0799a463093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:05.619875 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.619839 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac7a720-aaab-4619-b3be-a0799a463093-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4\" (UID: \"8ac7a720-aaab-4619-b3be-a0799a463093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:05.631219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.631187 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8"] Oct 31 21:42:05.720874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.720837 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac7a720-aaab-4619-b3be-a0799a463093-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4\" (UID: \"8ac7a720-aaab-4619-b3be-a0799a463093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:05.720874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.720880 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbpg2\" (UniqueName: \"kubernetes.io/projected/00581fdc-a8f7-4031-9d69-27f2b1c35307-kube-api-access-xbpg2\") pod \"payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8\" (UID: \"00581fdc-a8f7-4031-9d69-27f2b1c35307\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:05.721151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.720935 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxzph\" (UniqueName: \"kubernetes.io/projected/8ac7a720-aaab-4619-b3be-a0799a463093-kube-api-access-pxzph\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4\" (UID: \"8ac7a720-aaab-4619-b3be-a0799a463093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:05.721151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.720993 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00581fdc-a8f7-4031-9d69-27f2b1c35307-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8\" (UID: \"00581fdc-a8f7-4031-9d69-27f2b1c35307\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:05.721836 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.721814 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac7a720-aaab-4619-b3be-a0799a463093-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4\" (UID: \"8ac7a720-aaab-4619-b3be-a0799a463093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:05.744171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.744087 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxzph\" (UniqueName: \"kubernetes.io/projected/8ac7a720-aaab-4619-b3be-a0799a463093-kube-api-access-pxzph\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4\" (UID: \"8ac7a720-aaab-4619-b3be-a0799a463093\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:05.812421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.812373 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:05.822961 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.822593 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbpg2\" (UniqueName: \"kubernetes.io/projected/00581fdc-a8f7-4031-9d69-27f2b1c35307-kube-api-access-xbpg2\") pod \"payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8\" (UID: \"00581fdc-a8f7-4031-9d69-27f2b1c35307\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:05.822961 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.822748 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00581fdc-a8f7-4031-9d69-27f2b1c35307-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8\" (UID: \"00581fdc-a8f7-4031-9d69-27f2b1c35307\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:05.823845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.823801 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00581fdc-a8f7-4031-9d69-27f2b1c35307-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8\" (UID: \"00581fdc-a8f7-4031-9d69-27f2b1c35307\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:05.849120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.846998 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbpg2\" (UniqueName: \"kubernetes.io/projected/00581fdc-a8f7-4031-9d69-27f2b1c35307-kube-api-access-xbpg2\") pod \"payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8\" (UID: \"00581fdc-a8f7-4031-9d69-27f2b1c35307\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:05.928508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.928431 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:05.935680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.935646 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx"] Oct 31 21:42:05.935920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:05.935887 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" podUID="e4400c6f-3abb-4ca9-bbfe-7e0fced98609" containerName="maas-api" containerID="cri-o://56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78" gracePeriod=30 Oct 31 21:42:06.176101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.176055 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4"] Oct 31 21:42:06.181751 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:42:06.181700 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac7a720_aaab_4619_b3be_a0799a463093.slice/crio-8674c2cd7881c30f0d954fe4eb192109c882d055a906d4130b42ea758fb6da25 WatchSource:0}: Error finding container 8674c2cd7881c30f0d954fe4eb192109c882d055a906d4130b42ea758fb6da25: Status 404 returned error can't find the container with id 8674c2cd7881c30f0d954fe4eb192109c882d055a906d4130b42ea758fb6da25 Oct 31 21:42:06.293560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.293517 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8"] Oct 31 21:42:06.294667 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:42:06.294623 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00581fdc_a8f7_4031_9d69_27f2b1c35307.slice/crio-aff1d227a14a4b3a2ce97746b0cff6a88bd9eb6b4f0283a2e816754032ab863e WatchSource:0}: Error finding container aff1d227a14a4b3a2ce97746b0cff6a88bd9eb6b4f0283a2e816754032ab863e: Status 404 returned error can't find the container with id aff1d227a14a4b3a2ce97746b0cff6a88bd9eb6b4f0283a2e816754032ab863e Oct 31 21:42:06.321628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.321604 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:42:06.434627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.434592 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-openshift-service-ca\") pod \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " Oct 31 21:42:06.434799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.434688 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-kube-api-access-cvsbg\" (UniqueName: \"kubernetes.io/projected/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-kube-api-access-cvsbg\") pod \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " Oct 31 21:42:06.434799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.434716 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-maas-api-tls\") pod \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " Oct 31 21:42:06.434799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.434767 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-metrics-certs\") pod \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\" (UID: \"e4400c6f-3abb-4ca9-bbfe-7e0fced98609\") " Oct 31 21:42:06.435010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.434980 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-openshift-service-ca" pod "e4400c6f-3abb-4ca9-bbfe-7e0fced98609" (UID: "e4400c6f-3abb-4ca9-bbfe-7e0fced98609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:06.435239 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.435216 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:06.436939 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.436915 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-metrics-certs" pod "e4400c6f-3abb-4ca9-bbfe-7e0fced98609" (UID: "e4400c6f-3abb-4ca9-bbfe-7e0fced98609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:42:06.437415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.437393 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-kube-api-access-cvsbg" pod "e4400c6f-3abb-4ca9-bbfe-7e0fced98609" (UID: "e4400c6f-3abb-4ca9-bbfe-7e0fced98609"). InnerVolumeSpecName "kube-api-access-cvsbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:06.437502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.437418 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-maas-api-tls" pod "e4400c6f-3abb-4ca9-bbfe-7e0fced98609" (UID: "e4400c6f-3abb-4ca9-bbfe-7e0fced98609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:42:06.536659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.536500 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvsbg\" (UniqueName: \"kubernetes.io/projected/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-kube-api-access-cvsbg\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:06.536659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.536542 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:06.536659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.536556 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4400c6f-3abb-4ca9-bbfe-7e0fced98609-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:06.595126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.595004 5188 generic.go:356] "Generic (PLEG): container finished" podID="e4400c6f-3abb-4ca9-bbfe-7e0fced98609" containerID="56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78" exitCode=0 Oct 31 21:42:06.595126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.595049 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" event={"ID":"e4400c6f-3abb-4ca9-bbfe-7e0fced98609","Type":"ContainerDied","Data":"56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78"} Oct 31 21:42:06.595126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.595083 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" Oct 31 21:42:06.595126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.595116 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx" event={"ID":"e4400c6f-3abb-4ca9-bbfe-7e0fced98609","Type":"ContainerDied","Data":"294d6121e0df3d805e8c6b71f6cfaec39ed3857caa6655307c8725478869b3ee"} Oct 31 21:42:06.595602 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.595144 5188 scope.go:122] "RemoveContainer" containerID="56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78" Oct 31 21:42:06.601112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.601077 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" event={"ID":"00581fdc-a8f7-4031-9d69-27f2b1c35307","Type":"ContainerStarted","Data":"8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556"} Oct 31 21:42:06.602476 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.602446 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" event={"ID":"00581fdc-a8f7-4031-9d69-27f2b1c35307","Type":"ContainerStarted","Data":"aff1d227a14a4b3a2ce97746b0cff6a88bd9eb6b4f0283a2e816754032ab863e"} Oct 31 21:42:06.602596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.602539 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:06.604091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.603399 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" event={"ID":"8ac7a720-aaab-4619-b3be-a0799a463093","Type":"ContainerStarted","Data":"1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31"} Oct 31 21:42:06.604091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.603450 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" event={"ID":"8ac7a720-aaab-4619-b3be-a0799a463093","Type":"ContainerStarted","Data":"8674c2cd7881c30f0d954fe4eb192109c882d055a906d4130b42ea758fb6da25"} Oct 31 21:42:06.604091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.603491 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:06.612070 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.612037 5188 scope.go:122] "RemoveContainer" containerID="56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78" Oct 31 21:42:06.612450 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:06.612424 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78\": container with ID starting with 56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78 not found: ID does not exist" containerID="56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78" Oct 31 21:42:06.612523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.612463 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78"} err="failed to get container status \"56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78\": rpc error: code = NotFound desc = could not find container \"56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78\": container with ID starting with 56c677cd024dbca98a377002084381e210d4a4f3955bf80ae343d3c5829f3e78 not found: ID does not exist" Oct 31 21:42:06.647952 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.647919 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx"] Oct 31 21:42:06.668377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.655020 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9bbfc5-78c6c8dcb5-s6hwx"] Oct 31 21:42:06.691185 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:06.691122 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" podStartSLOduration=1.6910993090000002 podStartE2EDuration="1.691099309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:06.688291422 +0000 UTC m=+1573.381248816" watchObservedRunningTime="2026-10-31 21:42:06.691099309 +0000 UTC m=+1573.384056702" Oct 31 21:42:07.485033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:07.484998 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4400c6f-3abb-4ca9-bbfe-7e0fced98609" path="/var/lib/kubelet/pods/e4400c6f-3abb-4ca9-bbfe-7e0fced98609/volumes" Oct 31 21:42:08.284670 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.284615 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" podStartSLOduration=3.284600323 podStartE2EDuration="3.284600323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:06.728724895 +0000 UTC m=+1573.421682289" watchObservedRunningTime="2026-10-31 21:42:08.284600323 +0000 UTC m=+1574.977557743" Oct 31 21:42:08.285177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.285160 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn"] Oct 31 21:42:08.285806 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.285792 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e4400c6f-3abb-4ca9-bbfe-7e0fced98609" containerName="maas-api" Oct 31 21:42:08.285855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.285807 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4400c6f-3abb-4ca9-bbfe-7e0fced98609" containerName="maas-api" Oct 31 21:42:08.285912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.285903 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4400c6f-3abb-4ca9-bbfe-7e0fced98609" containerName="maas-api" Oct 31 21:42:08.291472 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.291442 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.312611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.312576 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn"] Oct 31 21:42:08.314556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.314508 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:42:08.314709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.314610 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:42:08.364498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.364451 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpnvs\" (UniqueName: \"kubernetes.io/projected/e626eaef-b437-4c91-8726-7de55755ca50-kube-api-access-tpnvs\") pod \"maas-api-revoke-sub-d7b302ad-qqtdn\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.364730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.364642 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b302ad-qqtdn\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.364809 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.364783 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b302ad-qqtdn\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.466207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.466164 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b302ad-qqtdn\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.466469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.466309 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b302ad-qqtdn\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.466469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.466402 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpnvs\" (UniqueName: \"kubernetes.io/projected/e626eaef-b437-4c91-8726-7de55755ca50-kube-api-access-tpnvs\") pod \"maas-api-revoke-sub-d7b302ad-qqtdn\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.466913 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.466887 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7b302ad-qqtdn\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.468741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.468717 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7b302ad-qqtdn\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.488102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.488066 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpnvs\" (UniqueName: \"kubernetes.io/projected/e626eaef-b437-4c91-8726-7de55755ca50-kube-api-access-tpnvs\") pod \"maas-api-revoke-sub-d7b302ad-qqtdn\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.595922 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.595832 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl"] Oct 31 21:42:08.601710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.601679 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.607675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.607642 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:08.620752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.620717 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl"] Oct 31 21:42:08.624405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.620945 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-6d6576-serving-cert\"" Oct 31 21:42:08.669355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.668768 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.669355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.668853 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38841d38-152d-4315-a8b8-7eba09d98ebf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.669355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.668950 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6tn\" (UniqueName: \"kubernetes.io/projected/38841d38-152d-4315-a8b8-7eba09d98ebf-kube-api-access-xw6tn\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.669355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.669087 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-metrics-certs\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.770550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.770510 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-metrics-certs\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.770773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.770622 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.770773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.770681 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38841d38-152d-4315-a8b8-7eba09d98ebf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.770903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.770772 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6tn\" (UniqueName: \"kubernetes.io/projected/38841d38-152d-4315-a8b8-7eba09d98ebf-kube-api-access-xw6tn\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.771748 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.771719 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38841d38-152d-4315-a8b8-7eba09d98ebf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.773465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.773440 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-metrics-certs\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.774295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.773965 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.792613 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.792585 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6tn\" (UniqueName: \"kubernetes.io/projected/38841d38-152d-4315-a8b8-7eba09d98ebf-kube-api-access-xw6tn\") pod \"maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.817898 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.817858 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp"] Oct 31 21:42:08.824181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.824156 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:08.843652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.840515 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1d8f7b-dockercfg-pb7cx\"" Oct 31 21:42:08.843652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.840604 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1d8f7b\"" Oct 31 21:42:08.843652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.843518 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp"] Oct 31 21:42:08.872366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.872314 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkm4g\" (UniqueName: \"kubernetes.io/projected/2e83a0d0-2433-4697-8779-6334653be0d9-kube-api-access-rkm4g\") pod \"payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp\" (UID: \"2e83a0d0-2433-4697-8779-6334653be0d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:08.872691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.872667 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e83a0d0-2433-4697-8779-6334653be0d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp\" (UID: \"2e83a0d0-2433-4697-8779-6334653be0d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:08.906597 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.905794 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk"] Oct 31 21:42:08.918152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.918105 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:08.926997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.926960 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:08.945005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.944969 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk"] Oct 31 21:42:08.975897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.974224 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e83a0d0-2433-4697-8779-6334653be0d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp\" (UID: \"2e83a0d0-2433-4697-8779-6334653be0d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:08.975897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.974338 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc824cf-e9da-4003-be08-c139b49d1652-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk\" (UID: \"6dc824cf-e9da-4003-be08-c139b49d1652\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:08.975897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.974390 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8nj\" (UniqueName: \"kubernetes.io/projected/6dc824cf-e9da-4003-be08-c139b49d1652-kube-api-access-lh8nj\") pod \"payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk\" (UID: \"6dc824cf-e9da-4003-be08-c139b49d1652\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:08.975897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.974539 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkm4g\" (UniqueName: \"kubernetes.io/projected/2e83a0d0-2433-4697-8779-6334653be0d9-kube-api-access-rkm4g\") pod \"payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp\" (UID: \"2e83a0d0-2433-4697-8779-6334653be0d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:08.975897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.975807 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e83a0d0-2433-4697-8779-6334653be0d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp\" (UID: \"2e83a0d0-2433-4697-8779-6334653be0d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:08.985825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:08.985781 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn"] Oct 31 21:42:09.007709 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.007679 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkm4g\" (UniqueName: \"kubernetes.io/projected/2e83a0d0-2433-4697-8779-6334653be0d9-kube-api-access-rkm4g\") pod \"payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp\" (UID: \"2e83a0d0-2433-4697-8779-6334653be0d9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:09.080095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.078317 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc824cf-e9da-4003-be08-c139b49d1652-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk\" (UID: \"6dc824cf-e9da-4003-be08-c139b49d1652\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:09.080095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.078405 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8nj\" (UniqueName: \"kubernetes.io/projected/6dc824cf-e9da-4003-be08-c139b49d1652-kube-api-access-lh8nj\") pod \"payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk\" (UID: \"6dc824cf-e9da-4003-be08-c139b49d1652\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:09.080095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.079784 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc824cf-e9da-4003-be08-c139b49d1652-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk\" (UID: \"6dc824cf-e9da-4003-be08-c139b49d1652\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:09.105443 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.105369 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8nj\" (UniqueName: \"kubernetes.io/projected/6dc824cf-e9da-4003-be08-c139b49d1652-kube-api-access-lh8nj\") pod \"payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk\" (UID: \"6dc824cf-e9da-4003-be08-c139b49d1652\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:09.141156 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.141006 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:09.255132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.255094 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:09.337379 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.337031 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl"] Oct 31 21:42:09.355567 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:42:09.355511 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38841d38_152d_4315_a8b8_7eba09d98ebf.slice/crio-b36b84fc2b8fd889321a16e7611c6dfbf2246175163955fd5c31b79a79024441 WatchSource:0}: Error finding container b36b84fc2b8fd889321a16e7611c6dfbf2246175163955fd5c31b79a79024441: Status 404 returned error can't find the container with id b36b84fc2b8fd889321a16e7611c6dfbf2246175163955fd5c31b79a79024441 Oct 31 21:42:09.519566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.519530 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp"] Oct 31 21:42:09.587038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.586985 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk"] Oct 31 21:42:09.589062 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:42:09.589024 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc824cf_e9da_4003_be08_c139b49d1652.slice/crio-3b3682f1c1b0c8c07c2cad16c73057f6cfd39dc84228c00c902ac311cb4189b6 WatchSource:0}: Error finding container 3b3682f1c1b0c8c07c2cad16c73057f6cfd39dc84228c00c902ac311cb4189b6: Status 404 returned error can't find the container with id 3b3682f1c1b0c8c07c2cad16c73057f6cfd39dc84228c00c902ac311cb4189b6 Oct 31 21:42:09.638529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.638490 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" event={"ID":"2e83a0d0-2433-4697-8779-6334653be0d9","Type":"ContainerStarted","Data":"ef44d366da37811eaee6960c5671a3528d857f97c17b36ee06a83743b4730f87"} Oct 31 21:42:09.638689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.638553 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" event={"ID":"2e83a0d0-2433-4697-8779-6334653be0d9","Type":"ContainerStarted","Data":"55a3ca74730766db21f69f0a5f7bcc6b48e9afb00728985cd15649e1b3048ae1"} Oct 31 21:42:09.638689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.638608 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:09.639978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.639950 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" event={"ID":"38841d38-152d-4315-a8b8-7eba09d98ebf","Type":"ContainerStarted","Data":"b36b84fc2b8fd889321a16e7611c6dfbf2246175163955fd5c31b79a79024441"} Oct 31 21:42:09.641807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.641747 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" event={"ID":"e626eaef-b437-4c91-8726-7de55755ca50","Type":"ContainerStarted","Data":"4dbcb05cd8cc89dc94e881bd776e67e4e87304871315936cc786b734166db598"} Oct 31 21:42:09.641807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.641799 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" event={"ID":"e626eaef-b437-4c91-8726-7de55755ca50","Type":"ContainerStarted","Data":"09af694ff31ae77d417bb18ebe149f87e4706ec1daa91eab64186607be870543"} Oct 31 21:42:09.643067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.643043 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" event={"ID":"6dc824cf-e9da-4003-be08-c139b49d1652","Type":"ContainerStarted","Data":"3b3682f1c1b0c8c07c2cad16c73057f6cfd39dc84228c00c902ac311cb4189b6"} Oct 31 21:42:09.677279 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.677220 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" podStartSLOduration=1.67720038 podStartE2EDuration="1.67720038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:09.675464014 +0000 UTC m=+1576.368421410" watchObservedRunningTime="2026-10-31 21:42:09.67720038 +0000 UTC m=+1576.370157777" Oct 31 21:42:09.714575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:09.714505 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" podStartSLOduration=1.714489265 podStartE2EDuration="1.714489265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:09.710734524 +0000 UTC m=+1576.403691917" watchObservedRunningTime="2026-10-31 21:42:09.714489265 +0000 UTC m=+1576.407446801" Oct 31 21:42:10.652708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:10.652669 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" event={"ID":"6dc824cf-e9da-4003-be08-c139b49d1652","Type":"ContainerStarted","Data":"112f7fb59f2f688ae95c8766a7a5544487586785102200d1926432d07a983b8f"} Oct 31 21:42:10.653257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:10.653228 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:10.654944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:10.654917 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" event={"ID":"38841d38-152d-4315-a8b8-7eba09d98ebf","Type":"ContainerStarted","Data":"dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7"} Oct 31 21:42:10.655102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:10.654992 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:10.656954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:10.656927 5188 generic.go:356] "Generic (PLEG): container finished" podID="e626eaef-b437-4c91-8726-7de55755ca50" containerID="4dbcb05cd8cc89dc94e881bd776e67e4e87304871315936cc786b734166db598" exitCode=0 Oct 31 21:42:10.657048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:10.657008 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" event={"ID":"e626eaef-b437-4c91-8726-7de55755ca50","Type":"ContainerDied","Data":"4dbcb05cd8cc89dc94e881bd776e67e4e87304871315936cc786b734166db598"} Oct 31 21:42:10.693537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:10.693470 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" podStartSLOduration=2.6934524509999997 podStartE2EDuration="2.693452451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:10.688280582 +0000 UTC m=+1577.381237975" watchObservedRunningTime="2026-10-31 21:42:10.693452451 +0000 UTC m=+1577.386409843" Oct 31 21:42:10.725953 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:10.725889 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" podStartSLOduration=2.437557771 podStartE2EDuration="2.72587594s" totalImagesPullingTime="288.318169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:10.72539496 +0000 UTC m=+1577.418352358" watchObservedRunningTime="2026-10-31 21:42:10.72587594 +0000 UTC m=+1577.418833330" Oct 31 21:42:11.837731 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:11.837702 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:11.926607 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:11.926501 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e626eaef-b437-4c91-8726-7de55755ca50-kube-api-access-tpnvs\" (UniqueName: \"kubernetes.io/projected/e626eaef-b437-4c91-8726-7de55755ca50-kube-api-access-tpnvs\") pod \"e626eaef-b437-4c91-8726-7de55755ca50\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " Oct 31 21:42:11.926607 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:11.926566 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-openshift-ca\") pod \"e626eaef-b437-4c91-8726-7de55755ca50\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " Oct 31 21:42:11.926861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:11.926819 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-xks-ca\") pod \"e626eaef-b437-4c91-8726-7de55755ca50\" (UID: \"e626eaef-b437-4c91-8726-7de55755ca50\") " Oct 31 21:42:11.928745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:11.928692 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-openshift-ca" pod "e626eaef-b437-4c91-8726-7de55755ca50" (UID: "e626eaef-b437-4c91-8726-7de55755ca50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:11.929041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:11.929013 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e626eaef-b437-4c91-8726-7de55755ca50-kube-api-access-tpnvs" pod "e626eaef-b437-4c91-8726-7de55755ca50" (UID: "e626eaef-b437-4c91-8726-7de55755ca50"). InnerVolumeSpecName "kube-api-access-tpnvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:11.930463 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:11.930435 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-xks-ca" pod "e626eaef-b437-4c91-8726-7de55755ca50" (UID: "e626eaef-b437-4c91-8726-7de55755ca50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:42:12.028436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:12.028396 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:12.028436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:12.028434 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpnvs\" (UniqueName: \"kubernetes.io/projected/e626eaef-b437-4c91-8726-7de55755ca50-kube-api-access-tpnvs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:12.028436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:12.028446 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e626eaef-b437-4c91-8726-7de55755ca50-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:12.674873 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:12.674825 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" event={"ID":"e626eaef-b437-4c91-8726-7de55755ca50","Type":"ContainerDied","Data":"09af694ff31ae77d417bb18ebe149f87e4706ec1daa91eab64186607be870543"} Oct 31 21:42:12.674873 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:12.674854 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn" Oct 31 21:42:12.675113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:12.674891 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09af694ff31ae77d417bb18ebe149f87e4706ec1daa91eab64186607be870543" Oct 31 21:42:16.158559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.158513 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" podUID="f2610999-7b04-479d-9322-fd0dadd6d0ea" containerName="main" containerID="cri-o://b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf" gracePeriod=45 Oct 31 21:42:16.505786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.505752 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:42:16.607123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.607083 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-dshm\") pod \"f2610999-7b04-479d-9322-fd0dadd6d0ea\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " Oct 31 21:42:16.607348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.607137 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-tmp-dir\") pod \"f2610999-7b04-479d-9322-fd0dadd6d0ea\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " Oct 31 21:42:16.607348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.607246 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2610999-7b04-479d-9322-fd0dadd6d0ea-kube-api-access-l8ngt\" (UniqueName: \"kubernetes.io/projected/f2610999-7b04-479d-9322-fd0dadd6d0ea-kube-api-access-l8ngt\") pod \"f2610999-7b04-479d-9322-fd0dadd6d0ea\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " Oct 31 21:42:16.607495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.607434 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2610999-7b04-479d-9322-fd0dadd6d0ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2610999-7b04-479d-9322-fd0dadd6d0ea-tls-certs\") pod \"f2610999-7b04-479d-9322-fd0dadd6d0ea\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " Oct 31 21:42:16.607558 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.607513 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-model-cache\") pod \"f2610999-7b04-479d-9322-fd0dadd6d0ea\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " Oct 31 21:42:16.607650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.607615 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-home\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-home\") pod \"f2610999-7b04-479d-9322-fd0dadd6d0ea\" (UID: \"f2610999-7b04-479d-9322-fd0dadd6d0ea\") " Oct 31 21:42:16.607789 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.607756 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-tmp-dir" pod "f2610999-7b04-479d-9322-fd0dadd6d0ea" (UID: "f2610999-7b04-479d-9322-fd0dadd6d0ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:16.608115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.608065 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-home" pod "f2610999-7b04-479d-9322-fd0dadd6d0ea" (UID: "f2610999-7b04-479d-9322-fd0dadd6d0ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:16.608273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.608232 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-model-cache" pod "f2610999-7b04-479d-9322-fd0dadd6d0ea" (UID: "f2610999-7b04-479d-9322-fd0dadd6d0ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:16.608456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.608293 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:16.608456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.608314 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:16.610450 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.610418 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2610999-7b04-479d-9322-fd0dadd6d0ea-tls-certs" pod "f2610999-7b04-479d-9322-fd0dadd6d0ea" (UID: "f2610999-7b04-479d-9322-fd0dadd6d0ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:42:16.610556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.610458 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2610999-7b04-479d-9322-fd0dadd6d0ea-kube-api-access-l8ngt" pod "f2610999-7b04-479d-9322-fd0dadd6d0ea" (UID: "f2610999-7b04-479d-9322-fd0dadd6d0ea"). InnerVolumeSpecName "kube-api-access-l8ngt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:16.610556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.610495 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-dshm" pod "f2610999-7b04-479d-9322-fd0dadd6d0ea" (UID: "f2610999-7b04-479d-9322-fd0dadd6d0ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:16.672424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.672382 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:42:16.710231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.710195 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2610999-7b04-479d-9322-fd0dadd6d0ea-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:16.710231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.710228 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:16.710231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.710239 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2610999-7b04-479d-9322-fd0dadd6d0ea-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:16.710569 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.710252 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8ngt\" (UniqueName: \"kubernetes.io/projected/f2610999-7b04-479d-9322-fd0dadd6d0ea-kube-api-access-l8ngt\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:16.715216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.715179 5188 generic.go:356] "Generic (PLEG): container finished" podID="f2610999-7b04-479d-9322-fd0dadd6d0ea" containerID="b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf" exitCode=0 Oct 31 21:42:16.715410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.715234 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" Oct 31 21:42:16.715410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.715274 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" event={"ID":"f2610999-7b04-479d-9322-fd0dadd6d0ea","Type":"ContainerDied","Data":"b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf"} Oct 31 21:42:16.715410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.715367 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q" event={"ID":"f2610999-7b04-479d-9322-fd0dadd6d0ea","Type":"ContainerDied","Data":"d6475fbe32c19ac3123559257e10b62e88066d50578861fc15ce5a2f456106d7"} Oct 31 21:42:16.715410 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.715397 5188 scope.go:122] "RemoveContainer" containerID="b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf" Oct 31 21:42:16.740150 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.740111 5188 scope.go:122] "RemoveContainer" containerID="b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf" Oct 31 21:42:16.740612 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:16.740580 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf\": container with ID starting with b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf not found: ID does not exist" containerID="b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf" Oct 31 21:42:16.740721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.740625 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf"} err="failed to get container status \"b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf\": rpc error: code = NotFound desc = could not find container \"b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf\": container with ID starting with b20522fdc2c8a2ce7fe2540de08219bd9c875b4d994c2779857153fb14048faf not found: ID does not exist" Oct 31 21:42:16.793386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.789650 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q"] Oct 31 21:42:16.794925 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:16.794886 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/tenantcfg-model-c744c0-kserve-dcdfc486f-7676q"] Oct 31 21:42:17.475001 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:17.474965 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2610999-7b04-479d-9322-fd0dadd6d0ea" path="/var/lib/kubelet/pods/f2610999-7b04-479d-9322-fd0dadd6d0ea/volumes" Oct 31 21:42:17.617628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:17.617598 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:18.633044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:18.633016 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:20.660860 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:20.660811 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:42:21.667111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.667075 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v"] Oct 31 21:42:21.668148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.668070 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f2610999-7b04-479d-9322-fd0dadd6d0ea" containerName="main" Oct 31 21:42:21.668148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.668099 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2610999-7b04-479d-9322-fd0dadd6d0ea" containerName="main" Oct 31 21:42:21.668148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.668121 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e626eaef-b437-4c91-8726-7de55755ca50" containerName="revoke-keys" Oct 31 21:42:21.668148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.668130 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e626eaef-b437-4c91-8726-7de55755ca50" containerName="revoke-keys" Oct 31 21:42:21.668395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.668335 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2610999-7b04-479d-9322-fd0dadd6d0ea" containerName="main" Oct 31 21:42:21.668395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.668354 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e626eaef-b437-4c91-8726-7de55755ca50" containerName="revoke-keys" Oct 31 21:42:21.673466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.673438 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:42:21.673627 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.673564 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.690877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.690829 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v"] Oct 31 21:42:21.766768 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.766725 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d43b66-serving-cert\"" Oct 31 21:42:21.782908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.782849 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd05979e-2766-4060-bb1b-38eeb0faf012-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.782908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.782918 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.783178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.783037 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54mk\" (UniqueName: \"kubernetes.io/projected/bd05979e-2766-4060-bb1b-38eeb0faf012-kube-api-access-z54mk\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.783280 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.783257 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-metrics-certs\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.884739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.884698 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd05979e-2766-4060-bb1b-38eeb0faf012-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.884739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.884742 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.884995 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.884775 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54mk\" (UniqueName: \"kubernetes.io/projected/bd05979e-2766-4060-bb1b-38eeb0faf012-kube-api-access-z54mk\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.884995 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.884932 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-metrics-certs\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.885682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.885650 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd05979e-2766-4060-bb1b-38eeb0faf012-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.887439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.887404 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.887550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.887450 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-metrics-certs\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.910152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.910111 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54mk\" (UniqueName: \"kubernetes.io/projected/bd05979e-2766-4060-bb1b-38eeb0faf012-kube-api-access-z54mk\") pod \"maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:21.991598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:21.991496 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:22.319008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:22.318960 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v"] Oct 31 21:42:22.331867 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:42:22.331824 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd05979e_2766_4060_bb1b_38eeb0faf012.slice/crio-3bc681a8b9ffce8f85bc94e2db1a3f2544d21db53b86ba4766eb6dbae5c161e4 WatchSource:0}: Error finding container 3bc681a8b9ffce8f85bc94e2db1a3f2544d21db53b86ba4766eb6dbae5c161e4: Status 404 returned error can't find the container with id 3bc681a8b9ffce8f85bc94e2db1a3f2544d21db53b86ba4766eb6dbae5c161e4 Oct 31 21:42:22.803370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:22.797770 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" event={"ID":"bd05979e-2766-4060-bb1b-38eeb0faf012","Type":"ContainerStarted","Data":"ed9d05e143e92120eb6a463d5855260c1f599c8edcaf3733ccdeadfa0b10400e"} Oct 31 21:42:22.803370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:22.797833 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" event={"ID":"bd05979e-2766-4060-bb1b-38eeb0faf012","Type":"ContainerStarted","Data":"3bc681a8b9ffce8f85bc94e2db1a3f2544d21db53b86ba4766eb6dbae5c161e4"} Oct 31 21:42:22.803370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:22.797876 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:22.839409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:22.839343 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" podStartSLOduration=1.511836392 podStartE2EDuration="1.839305941s" totalImagesPullingTime="327.469549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:22.837938378 +0000 UTC m=+1589.530895770" watchObservedRunningTime="2026-10-31 21:42:22.839305941 +0000 UTC m=+1589.532263331" Oct 31 21:42:23.009656 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:23.009526 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j"] Oct 31 21:42:23.009849 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:23.009819 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" containerID="cri-o://cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe" gracePeriod=30 Oct 31 21:42:23.021042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:23.014779 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk"] Oct 31 21:42:23.021042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:23.015040 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" containerID="cri-o://3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d" gracePeriod=30 Oct 31 21:42:26.879138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:26.879075 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8"] Oct 31 21:42:26.885217 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:26.885185 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:26.907149 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:26.907113 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1d8f7b-serving-cert\"" Oct 31 21:42:26.911638 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:26.911583 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8"] Oct 31 21:42:26.987520 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:26.987482 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-metrics-certs\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:26.987762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:26.987532 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16bdeb02-6f38-4b5f-9bc7-feaba894e996-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:26.987762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:26.987665 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbwr\" (UniqueName: \"kubernetes.io/projected/16bdeb02-6f38-4b5f-9bc7-feaba894e996-kube-api-access-xjbwr\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:26.987909 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:26.987892 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.089741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.089688 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.089957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.089850 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-metrics-certs\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.089957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.089875 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16bdeb02-6f38-4b5f-9bc7-feaba894e996-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.089957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.089899 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbwr\" (UniqueName: \"kubernetes.io/projected/16bdeb02-6f38-4b5f-9bc7-feaba894e996-kube-api-access-xjbwr\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.091128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.091094 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16bdeb02-6f38-4b5f-9bc7-feaba894e996-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.096014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.095974 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.099061 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.099027 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-metrics-certs\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.115810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.113855 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbwr\" (UniqueName: \"kubernetes.io/projected/16bdeb02-6f38-4b5f-9bc7-feaba894e996-kube-api-access-xjbwr\") pod \"maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.212895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.212779 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:27.532044 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.532011 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8"] Oct 31 21:42:27.534160 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:42:27.534117 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bdeb02_6f38_4b5f_9bc7_feaba894e996.slice/crio-3a2acf2c5b19d573fde070d56a1017f726c37136a6920a225275f09613621088 WatchSource:0}: Error finding container 3a2acf2c5b19d573fde070d56a1017f726c37136a6920a225275f09613621088: Status 404 returned error can't find the container with id 3a2acf2c5b19d573fde070d56a1017f726c37136a6920a225275f09613621088 Oct 31 21:42:27.842947 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:27.842859 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" event={"ID":"16bdeb02-6f38-4b5f-9bc7-feaba894e996","Type":"ContainerStarted","Data":"3a2acf2c5b19d573fde070d56a1017f726c37136a6920a225275f09613621088"} Oct 31 21:42:28.343071 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.343035 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:42:28.392160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.392077 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:42:28.415593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.414066 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-data\") pod \"7954fda0-ffa6-426f-b131-2b09ec6de759\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " Oct 31 21:42:28.415593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.414225 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-podinfo\") pod \"7954fda0-ffa6-426f-b131-2b09ec6de759\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " Oct 31 21:42:28.417205 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.417167 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-data" pod "7954fda0-ffa6-426f-b131-2b09ec6de759" (UID: "7954fda0-ffa6-426f-b131-2b09ec6de759"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.417393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.417262 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-token\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-token\") pod \"7954fda0-ffa6-426f-b131-2b09ec6de759\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " Oct 31 21:42:28.417393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.417298 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-envoy\") pod \"7954fda0-ffa6-426f-b131-2b09ec6de759\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " Oct 31 21:42:28.417513 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.417498 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-kube-api-access-f2hrj\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-kube-api-access-f2hrj\") pod \"7954fda0-ffa6-426f-b131-2b09ec6de759\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " Oct 31 21:42:28.417578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.417564 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-certs\") pod \"7954fda0-ffa6-426f-b131-2b09ec6de759\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " Oct 31 21:42:28.417650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.417605 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-credential-socket\") pod \"7954fda0-ffa6-426f-b131-2b09ec6de759\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " Oct 31 21:42:28.417701 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.417666 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7954fda0-ffa6-426f-b131-2b09ec6de759-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7954fda0-ffa6-426f-b131-2b09ec6de759-istiod-ca-cert\") pod \"7954fda0-ffa6-426f-b131-2b09ec6de759\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " Oct 31 21:42:28.417750 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.417724 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-socket\") pod \"7954fda0-ffa6-426f-b131-2b09ec6de759\" (UID: \"7954fda0-ffa6-426f-b131-2b09ec6de759\") " Oct 31 21:42:28.418298 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.418202 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-podinfo" pod "7954fda0-ffa6-426f-b131-2b09ec6de759" (UID: "7954fda0-ffa6-426f-b131-2b09ec6de759"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:42:28.418701 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.418680 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.418806 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.418704 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.418865 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.418841 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-credential-socket" pod "7954fda0-ffa6-426f-b131-2b09ec6de759" (UID: "7954fda0-ffa6-426f-b131-2b09ec6de759"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.419060 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.419007 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-certs" pod "7954fda0-ffa6-426f-b131-2b09ec6de759" (UID: "7954fda0-ffa6-426f-b131-2b09ec6de759"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.419130 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.419092 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-socket" pod "7954fda0-ffa6-426f-b131-2b09ec6de759" (UID: "7954fda0-ffa6-426f-b131-2b09ec6de759"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.419178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.419032 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7954fda0-ffa6-426f-b131-2b09ec6de759-istiod-ca-cert" pod "7954fda0-ffa6-426f-b131-2b09ec6de759" (UID: "7954fda0-ffa6-426f-b131-2b09ec6de759"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:28.421370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.421282 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-kube-api-access-f2hrj" pod "7954fda0-ffa6-426f-b131-2b09ec6de759" (UID: "7954fda0-ffa6-426f-b131-2b09ec6de759"). InnerVolumeSpecName "kube-api-access-f2hrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:28.422499 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.422462 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-envoy" pod "7954fda0-ffa6-426f-b131-2b09ec6de759" (UID: "7954fda0-ffa6-426f-b131-2b09ec6de759"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.423553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.423506 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-token" pod "7954fda0-ffa6-426f-b131-2b09ec6de759" (UID: "7954fda0-ffa6-426f-b131-2b09ec6de759"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:28.519791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.519743 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-certs\") pod \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " Oct 31 21:42:28.519791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.519795 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-envoy\") pod \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " Oct 31 21:42:28.520128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.519827 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-kube-api-access-jgln5\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-kube-api-access-jgln5\") pod \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " Oct 31 21:42:28.520128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.519846 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-credential-socket\") pod \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " Oct 31 21:42:28.520128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.519866 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-data\") pod \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " Oct 31 21:42:28.520128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.519893 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-socket\") pod \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " Oct 31 21:42:28.520128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.519925 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-token\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-token\") pod \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " Oct 31 21:42:28.520128 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.519986 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istiod-ca-cert\") pod \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " Oct 31 21:42:28.520523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.520061 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-certs" pod "4021eab3-c6e4-47d6-93b3-1c2178324bc0" (UID: "4021eab3-c6e4-47d6-93b3-1c2178324bc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.520577 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.520506 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-podinfo\") pod \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\" (UID: \"4021eab3-c6e4-47d6-93b3-1c2178324bc0\") " Oct 31 21:42:28.520642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.520291 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-credential-socket" pod "4021eab3-c6e4-47d6-93b3-1c2178324bc0" (UID: "4021eab3-c6e4-47d6-93b3-1c2178324bc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.520642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.520426 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-data" pod "4021eab3-c6e4-47d6-93b3-1c2178324bc0" (UID: "4021eab3-c6e4-47d6-93b3-1c2178324bc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.520642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.520459 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istiod-ca-cert" pod "4021eab3-c6e4-47d6-93b3-1c2178324bc0" (UID: "4021eab3-c6e4-47d6-93b3-1c2178324bc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:28.520799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.520478 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-socket" pod "4021eab3-c6e4-47d6-93b3-1c2178324bc0" (UID: "4021eab3-c6e4-47d6-93b3-1c2178324bc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.521123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521099 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2hrj\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-kube-api-access-f2hrj\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521125 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521135 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521144 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7954fda0-ffa6-426f-b131-2b09ec6de759-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521154 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521162 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521170 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521177 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521185 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521192 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521200 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.521254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.521208 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7954fda0-ffa6-426f-b131-2b09ec6de759-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.522575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.522547 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-kube-api-access-jgln5" pod "4021eab3-c6e4-47d6-93b3-1c2178324bc0" (UID: "4021eab3-c6e4-47d6-93b3-1c2178324bc0"). InnerVolumeSpecName "kube-api-access-jgln5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:28.522779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.522757 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-envoy" pod "4021eab3-c6e4-47d6-93b3-1c2178324bc0" (UID: "4021eab3-c6e4-47d6-93b3-1c2178324bc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:42:28.522966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.522940 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-token" pod "4021eab3-c6e4-47d6-93b3-1c2178324bc0" (UID: "4021eab3-c6e4-47d6-93b3-1c2178324bc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:28.523041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.523016 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-podinfo" pod "4021eab3-c6e4-47d6-93b3-1c2178324bc0" (UID: "4021eab3-c6e4-47d6-93b3-1c2178324bc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:42:28.622647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.622598 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.622647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.622653 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgln5\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-kube-api-access-jgln5\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.622877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.622671 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.622877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.622680 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4021eab3-c6e4-47d6-93b3-1c2178324bc0-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:28.816214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.816107 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:42:28.854144 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.854104 5188 generic.go:356] "Generic (PLEG): container finished" podID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerID="cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe" exitCode=0 Oct 31 21:42:28.854417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.854175 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" Oct 31 21:42:28.854417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.854257 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" event={"ID":"7954fda0-ffa6-426f-b131-2b09ec6de759","Type":"ContainerDied","Data":"cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe"} Oct 31 21:42:28.854417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.854309 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j" event={"ID":"7954fda0-ffa6-426f-b131-2b09ec6de759","Type":"ContainerDied","Data":"05f65c8e1483f82ad73039c951929b06de885b3063e78ce5a57688146f64a4b0"} Oct 31 21:42:28.854417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.854380 5188 scope.go:122] "RemoveContainer" containerID="cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe" Oct 31 21:42:28.859406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.859371 5188 generic.go:356] "Generic (PLEG): container finished" podID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerID="3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d" exitCode=0 Oct 31 21:42:28.859552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.859426 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" Oct 31 21:42:28.859552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.859419 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" event={"ID":"4021eab3-c6e4-47d6-93b3-1c2178324bc0","Type":"ContainerDied","Data":"3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d"} Oct 31 21:42:28.859727 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.859563 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk" event={"ID":"4021eab3-c6e4-47d6-93b3-1c2178324bc0","Type":"ContainerDied","Data":"7bda7e2d476ab26f91ff943f371cbba8463711cf9e7edc5e403129182118321b"} Oct 31 21:42:28.861837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.861798 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" event={"ID":"16bdeb02-6f38-4b5f-9bc7-feaba894e996","Type":"ContainerStarted","Data":"95acb131353244a30ae96947f609ec714dcc048790c70318fcca318e829499ee"} Oct 31 21:42:28.862232 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.862178 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:28.875579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.875539 5188 scope.go:122] "RemoveContainer" containerID="cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe" Oct 31 21:42:28.876051 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:28.875992 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe\": container with ID starting with cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe not found: ID does not exist" containerID="cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe" Oct 31 21:42:28.876051 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.876034 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe"} err="failed to get container status \"cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe\": rpc error: code = NotFound desc = could not find container \"cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe\": container with ID starting with cab4846b861da23b55fb80129e6324056a354f498fee7fcd0890ebac503401fe not found: ID does not exist" Oct 31 21:42:28.876218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.876061 5188 scope.go:122] "RemoveContainer" containerID="3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d" Oct 31 21:42:28.890556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.890530 5188 scope.go:122] "RemoveContainer" containerID="3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d" Oct 31 21:42:28.890953 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:28.890929 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d\": container with ID starting with 3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d not found: ID does not exist" containerID="3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d" Oct 31 21:42:28.891037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.890965 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d"} err="failed to get container status \"3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d\": rpc error: code = NotFound desc = could not find container \"3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d\": container with ID starting with 3add19fa3097d991a7566e6fd5b637475fe4ef60417d51548d40771e313ede1d not found: ID does not exist" Oct 31 21:42:28.908807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.908769 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j"] Oct 31 21:42:28.919282 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.919245 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-cts4j"] Oct 31 21:42:28.971575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:28.971497 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" podStartSLOduration=2.643818477 podStartE2EDuration="2.971477942s" totalImagesPullingTime="327.659465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:28.962602862 +0000 UTC m=+1595.655560250" watchObservedRunningTime="2026-10-31 21:42:28.971477942 +0000 UTC m=+1595.664435338" Oct 31 21:42:29.002099 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:29.002060 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk"] Oct 31 21:42:29.015772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:29.015724 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-f9pgk"] Oct 31 21:42:29.474001 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:29.473919 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" path="/var/lib/kubelet/pods/4021eab3-c6e4-47d6-93b3-1c2178324bc0/volumes" Oct 31 21:42:29.474448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:29.474434 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" path="/var/lib/kubelet/pods/7954fda0-ffa6-426f-b131-2b09ec6de759/volumes" Oct 31 21:42:32.540849 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:32.540794 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:42:34.880332 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:34.880289 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:42:35.350301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.350195 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt"] Oct 31 21:42:35.350584 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.350482 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" podUID="d2104e85-d811-4b12-9628-226615b41f16" containerName="payload-processing" containerID="cri-o://5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589" gracePeriod=30 Oct 31 21:42:35.389716 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.376980 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p"] Oct 31 21:42:35.389716 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.377229 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" podUID="bd2efa7d-10e7-4ac2-a9ac-44f96f561445" containerName="payload-pre-processing" containerID="cri-o://17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a" gracePeriod=30 Oct 31 21:42:35.915980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.915899 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:42:35.921518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.921490 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:42:35.931975 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.931942 5188 generic.go:356] "Generic (PLEG): container finished" podID="d2104e85-d811-4b12-9628-226615b41f16" containerID="5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589" exitCode=0 Oct 31 21:42:35.932173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.931992 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" Oct 31 21:42:35.932173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.932026 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" event={"ID":"d2104e85-d811-4b12-9628-226615b41f16","Type":"ContainerDied","Data":"5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589"} Oct 31 21:42:35.932173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.932086 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt" event={"ID":"d2104e85-d811-4b12-9628-226615b41f16","Type":"ContainerDied","Data":"f7838c4573ed8082f5e2b2af1bd7e6a0a8739fbeb5a431fe890578ccf085aef4"} Oct 31 21:42:35.932173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.932111 5188 scope.go:122] "RemoveContainer" containerID="5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589" Oct 31 21:42:35.933965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.933944 5188 generic.go:356] "Generic (PLEG): container finished" podID="bd2efa7d-10e7-4ac2-a9ac-44f96f561445" containerID="17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a" exitCode=0 Oct 31 21:42:35.934067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.934009 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" Oct 31 21:42:35.934125 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.934013 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" event={"ID":"bd2efa7d-10e7-4ac2-a9ac-44f96f561445","Type":"ContainerDied","Data":"17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a"} Oct 31 21:42:35.934177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.934131 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p" event={"ID":"bd2efa7d-10e7-4ac2-a9ac-44f96f561445","Type":"ContainerDied","Data":"d443de8282233048bb26b18707dc6f820b241e1a1a6b6341f984ecd65379d554"} Oct 31 21:42:35.944579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.944555 5188 scope.go:122] "RemoveContainer" containerID="5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589" Oct 31 21:42:35.944950 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:35.944924 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589\": container with ID starting with 5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589 not found: ID does not exist" containerID="5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589" Oct 31 21:42:35.945024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.944960 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589"} err="failed to get container status \"5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589\": rpc error: code = NotFound desc = could not find container \"5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589\": container with ID starting with 5881325ce030de87df98477b3bfb51e1cdcd73d9a96616656b9ee40744c87589 not found: ID does not exist" Oct 31 21:42:35.945024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.944978 5188 scope.go:122] "RemoveContainer" containerID="17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a" Oct 31 21:42:35.947380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.947353 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-kube-api-access-5qlvd\" (UniqueName: \"kubernetes.io/projected/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-kube-api-access-5qlvd\") pod \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\" (UID: \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\") " Oct 31 21:42:35.947790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.947445 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-plugins-config-volume\") pod \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\" (UID: \"bd2efa7d-10e7-4ac2-a9ac-44f96f561445\") " Oct 31 21:42:35.947790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.947493 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2104e85-d811-4b12-9628-226615b41f16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2104e85-d811-4b12-9628-226615b41f16-plugins-config-volume\") pod \"d2104e85-d811-4b12-9628-226615b41f16\" (UID: \"d2104e85-d811-4b12-9628-226615b41f16\") " Oct 31 21:42:35.947790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.947525 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2104e85-d811-4b12-9628-226615b41f16-kube-api-access-wz4n4\" (UniqueName: \"kubernetes.io/projected/d2104e85-d811-4b12-9628-226615b41f16-kube-api-access-wz4n4\") pod \"d2104e85-d811-4b12-9628-226615b41f16\" (UID: \"d2104e85-d811-4b12-9628-226615b41f16\") " Oct 31 21:42:35.948342 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.947991 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-plugins-config-volume" pod "bd2efa7d-10e7-4ac2-a9ac-44f96f561445" (UID: "bd2efa7d-10e7-4ac2-a9ac-44f96f561445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:35.948342 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.948169 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:35.948467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.948426 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2104e85-d811-4b12-9628-226615b41f16-plugins-config-volume" pod "d2104e85-d811-4b12-9628-226615b41f16" (UID: "d2104e85-d811-4b12-9628-226615b41f16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:35.950006 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.949984 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-kube-api-access-5qlvd" pod "bd2efa7d-10e7-4ac2-a9ac-44f96f561445" (UID: "bd2efa7d-10e7-4ac2-a9ac-44f96f561445"). InnerVolumeSpecName "kube-api-access-5qlvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:35.950269 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.950249 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2104e85-d811-4b12-9628-226615b41f16-kube-api-access-wz4n4" pod "d2104e85-d811-4b12-9628-226615b41f16" (UID: "d2104e85-d811-4b12-9628-226615b41f16"). InnerVolumeSpecName "kube-api-access-wz4n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:35.958775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.958737 5188 scope.go:122] "RemoveContainer" containerID="17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a" Oct 31 21:42:35.959555 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:35.959521 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a\": container with ID starting with 17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a not found: ID does not exist" containerID="17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a" Oct 31 21:42:35.959715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:35.959561 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a"} err="failed to get container status \"17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a\": rpc error: code = NotFound desc = could not find container \"17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a\": container with ID starting with 17290f7e58ce3d1ee4e960f3800a4fa774a5a148feed89e49165c820581b3d8a not found: ID does not exist" Oct 31 21:42:36.049578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:36.049536 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qlvd\" (UniqueName: \"kubernetes.io/projected/bd2efa7d-10e7-4ac2-a9ac-44f96f561445-kube-api-access-5qlvd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:36.049578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:36.049571 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2104e85-d811-4b12-9628-226615b41f16-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:36.049578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:36.049583 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz4n4\" (UniqueName: \"kubernetes.io/projected/d2104e85-d811-4b12-9628-226615b41f16-kube-api-access-wz4n4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:36.283817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:36.283774 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt"] Oct 31 21:42:36.290529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:36.290485 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-6cd9b97bb-ppvzt"] Oct 31 21:42:36.322268 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:36.319483 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p"] Oct 31 21:42:36.328482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:36.328440 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-f9b755f5-5bk4p"] Oct 31 21:42:37.473682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.473639 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2efa7d-10e7-4ac2-a9ac-44f96f561445" path="/var/lib/kubelet/pods/bd2efa7d-10e7-4ac2-a9ac-44f96f561445/volumes" Oct 31 21:42:37.474234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.474213 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2104e85-d811-4b12-9628-226615b41f16" path="/var/lib/kubelet/pods/d2104e85-d811-4b12-9628-226615b41f16/volumes" Oct 31 21:42:37.835905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.835808 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh"] Oct 31 21:42:37.836729 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.836701 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" Oct 31 21:42:37.836928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.836904 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" Oct 31 21:42:37.837037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.836953 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bd2efa7d-10e7-4ac2-a9ac-44f96f561445" containerName="payload-pre-processing" Oct 31 21:42:37.837037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.836959 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2efa7d-10e7-4ac2-a9ac-44f96f561445" containerName="payload-pre-processing" Oct 31 21:42:37.837037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.836990 5188 cpu_manager.go:414] "containerMap: removing container" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" Oct 31 21:42:37.837037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.836995 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" Oct 31 21:42:37.837037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.837023 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d2104e85-d811-4b12-9628-226615b41f16" containerName="payload-processing" Oct 31 21:42:37.837037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.837028 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2104e85-d811-4b12-9628-226615b41f16" containerName="payload-processing" Oct 31 21:42:37.837240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.837169 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2efa7d-10e7-4ac2-a9ac-44f96f561445" containerName="payload-pre-processing" Oct 31 21:42:37.837240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.837184 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="4021eab3-c6e4-47d6-93b3-1c2178324bc0" containerName="istio-proxy" Oct 31 21:42:37.837240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.837194 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2104e85-d811-4b12-9628-226615b41f16" containerName="payload-processing" Oct 31 21:42:37.837240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.837201 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="7954fda0-ffa6-426f-b131-2b09ec6de759" containerName="istio-proxy" Oct 31 21:42:37.850960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.850925 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.876063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.876025 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bb9990-openshift-default-dockercfg-ljzpp\"" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.878064 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-certs\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.904300 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-credential-socket\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.904410 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-socket\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.904537 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istiod-ca-cert\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.904582 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-envoy\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.904629 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-podinfo\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.904717 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-token\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.904775 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5cc\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-kube-api-access-9n5cc\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.904861 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-data\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:37.906235 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:37.887847 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh"] Oct 31 21:42:38.013457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.013309 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-data\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.013457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.013407 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-certs\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.013457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.013446 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-credential-socket\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.013756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.013525 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-socket\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.013756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.013619 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istiod-ca-cert\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.013756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.013655 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-envoy\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.013756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.013692 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-podinfo\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.013942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.013763 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-token\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.013942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.013812 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5cc\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-kube-api-access-9n5cc\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.035939 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.035897 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-socket\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.045569 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.036498 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-data\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.045799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.036803 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-certs\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.046952 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.046919 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istiod-ca-cert\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.047651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.047605 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-credential-socket\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.053940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.053007 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-podinfo\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.053940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.053312 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-envoy\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.060992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.060922 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5cc\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-kube-api-access-9n5cc\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.061214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.061163 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-token\") pod \"e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.193393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.193351 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:38.526533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.526497 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh"] Oct 31 21:42:38.527253 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:42:38.527225 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ba141e_72e4_4a29_be5c_fa7c9dae6317.slice/crio-688a4e10c49b631ab86da329f2735c150475f3e42e9775e5a833b432f9716dc6 WatchSource:0}: Error finding container 688a4e10c49b631ab86da329f2735c150475f3e42e9775e5a833b432f9716dc6: Status 404 returned error can't find the container with id 688a4e10c49b631ab86da329f2735c150475f3e42e9775e5a833b432f9716dc6 Oct 31 21:42:38.530074 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.530037 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:42:38.530195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.530123 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:42:38.530195 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.530172 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:42:38.979640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.979564 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" event={"ID":"58ba141e-72e4-4a29-be5c-fa7c9dae6317","Type":"ContainerStarted","Data":"fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57"} Oct 31 21:42:38.979873 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:38.979672 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" event={"ID":"58ba141e-72e4-4a29-be5c-fa7c9dae6317","Type":"ContainerStarted","Data":"688a4e10c49b631ab86da329f2735c150475f3e42e9775e5a833b432f9716dc6"} Oct 31 21:42:39.023403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:39.023342 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" podStartSLOduration=2.023315444 podStartE2EDuration="2.023315444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:39.020563096 +0000 UTC m=+1605.713520512" watchObservedRunningTime="2026-10-31 21:42:39.023315444 +0000 UTC m=+1605.716272835" Oct 31 21:42:39.194034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:39.193979 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:39.202239 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:39.202207 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:39.989384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:39.989346 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:39.990361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:39.990310 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:42:41.914361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:41.914303 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk"] Oct 31 21:42:41.923009 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:41.922978 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:41.961347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:41.957112 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-80542b-serving-cert\"" Oct 31 21:42:42.011418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.011380 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk"] Oct 31 21:42:42.072592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.072211 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.072592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.072282 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgs9l\" (UniqueName: \"kubernetes.io/projected/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-kube-api-access-wgs9l\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.072592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.072313 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.072592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.072353 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-metrics-certs\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.173893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.173793 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgs9l\" (UniqueName: \"kubernetes.io/projected/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-kube-api-access-wgs9l\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.173893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.173868 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.173893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.173894 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-metrics-certs\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.174223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.174192 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.174901 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.174873 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.176552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.176528 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.176902 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.176882 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-metrics-certs\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.195684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.195642 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgs9l\" (UniqueName: \"kubernetes.io/projected/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-kube-api-access-wgs9l\") pod \"maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.273375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.273329 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:42.591029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:42.590995 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk"] Oct 31 21:42:42.594007 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:42:42.593974 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2e64cf_ece3_4520_9f7e_0e39d51919f5.slice/crio-b9bbb4b682f2a5f952db1fbd9c072ba3d97669525486e8d02e30e933990de3c9 WatchSource:0}: Error finding container b9bbb4b682f2a5f952db1fbd9c072ba3d97669525486e8d02e30e933990de3c9: Status 404 returned error can't find the container with id b9bbb4b682f2a5f952db1fbd9c072ba3d97669525486e8d02e30e933990de3c9 Oct 31 21:42:43.067756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:43.067713 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" event={"ID":"1c2e64cf-ece3-4520-9f7e-0e39d51919f5","Type":"ContainerStarted","Data":"563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802"} Oct 31 21:42:43.068334 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:43.067779 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" event={"ID":"1c2e64cf-ece3-4520-9f7e-0e39d51919f5","Type":"ContainerStarted","Data":"b9bbb4b682f2a5f952db1fbd9c072ba3d97669525486e8d02e30e933990de3c9"} Oct 31 21:42:43.068334 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:43.067882 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:43.107582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:43.107518 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" podStartSLOduration=1.754740652 podStartE2EDuration="2.10749902s" totalImagesPullingTime="352.758368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:43.105260598 +0000 UTC m=+1609.798217990" watchObservedRunningTime="2026-10-31 21:42:43.10749902 +0000 UTC m=+1609.800456413" Oct 31 21:42:43.905249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:43.905207 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x"] Oct 31 21:42:43.909769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:43.909740 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:42:43.924874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:43.924833 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bb9990\"" Oct 31 21:42:43.934364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:43.934319 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bb9990-dockercfg-gdl99\"" Oct 31 21:42:43.943357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:43.940886 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x"] Oct 31 21:42:44.005346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:44.002958 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfq7t\" (UniqueName: \"kubernetes.io/projected/265e13d7-4e0b-4989-96e9-779d4e19ba6c-kube-api-access-sfq7t\") pod \"payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x\" (UID: \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:42:44.005346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:44.003083 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265e13d7-4e0b-4989-96e9-779d4e19ba6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x\" (UID: \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:42:44.105471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:44.104653 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfq7t\" (UniqueName: \"kubernetes.io/projected/265e13d7-4e0b-4989-96e9-779d4e19ba6c-kube-api-access-sfq7t\") pod \"payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x\" (UID: \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:42:44.105471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:44.104750 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265e13d7-4e0b-4989-96e9-779d4e19ba6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x\" (UID: \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:42:44.106944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:44.106818 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265e13d7-4e0b-4989-96e9-779d4e19ba6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x\" (UID: \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:42:44.134586 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:44.129672 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfq7t\" (UniqueName: \"kubernetes.io/projected/265e13d7-4e0b-4989-96e9-779d4e19ba6c-kube-api-access-sfq7t\") pod \"payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x\" (UID: \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:42:44.228228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:44.228134 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:42:44.537066 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:44.537028 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x"] Oct 31 21:42:44.537998 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:42:44.537971 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265e13d7_4e0b_4989_96e9_779d4e19ba6c.slice/crio-417044a9260744734d87ce9dde2a8a78d361b9e933182fe407a9ede055551de7 WatchSource:0}: Error finding container 417044a9260744734d87ce9dde2a8a78d361b9e933182fe407a9ede055551de7: Status 404 returned error can't find the container with id 417044a9260744734d87ce9dde2a8a78d361b9e933182fe407a9ede055551de7 Oct 31 21:42:45.113660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:45.113598 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" event={"ID":"265e13d7-4e0b-4989-96e9-779d4e19ba6c","Type":"ContainerStarted","Data":"54e1b73e047e46e988147fe16c47182c4216b033953f8b72324f157ede091773"} Oct 31 21:42:45.114238 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:45.113696 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" event={"ID":"265e13d7-4e0b-4989-96e9-779d4e19ba6c","Type":"ContainerStarted","Data":"417044a9260744734d87ce9dde2a8a78d361b9e933182fe407a9ede055551de7"} Oct 31 21:42:45.114238 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:45.113843 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:42:45.151965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:45.151907 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" podStartSLOduration=2.151890291 podStartE2EDuration="2.151890291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:42:45.14945668 +0000 UTC m=+1611.842414074" watchObservedRunningTime="2026-10-31 21:42:45.151890291 +0000 UTC m=+1611.844847684" Oct 31 21:42:49.108048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:49.107990 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:42:49.564404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:49.564364 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8"] Oct 31 21:42:49.606509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:49.606416 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4"] Oct 31 21:42:54.564682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:54.564610 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" podUID="00581fdc-a8f7-4031-9d69-27f2b1c35307" containerName="payload-processing" containerID="cri-o://8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556" gracePeriod=25 Oct 31 21:42:54.607302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:54.607237 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" podUID="8ac7a720-aaab-4619-b3be-a0799a463093" containerName="payload-pre-processing" containerID="cri-o://1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31" gracePeriod=25 Oct 31 21:42:54.933140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:54.933110 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:54.986318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:54.986247 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:55.050467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.050425 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac7a720-aaab-4619-b3be-a0799a463093-kube-api-access-pxzph\" (UniqueName: \"kubernetes.io/projected/8ac7a720-aaab-4619-b3be-a0799a463093-kube-api-access-pxzph\") pod \"8ac7a720-aaab-4619-b3be-a0799a463093\" (UID: \"8ac7a720-aaab-4619-b3be-a0799a463093\") " Oct 31 21:42:55.050682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.050507 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00581fdc-a8f7-4031-9d69-27f2b1c35307-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00581fdc-a8f7-4031-9d69-27f2b1c35307-plugins-config-volume\") pod \"00581fdc-a8f7-4031-9d69-27f2b1c35307\" (UID: \"00581fdc-a8f7-4031-9d69-27f2b1c35307\") " Oct 31 21:42:55.050682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.050625 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac7a720-aaab-4619-b3be-a0799a463093-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac7a720-aaab-4619-b3be-a0799a463093-plugins-config-volume\") pod \"8ac7a720-aaab-4619-b3be-a0799a463093\" (UID: \"8ac7a720-aaab-4619-b3be-a0799a463093\") " Oct 31 21:42:55.050831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.050805 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00581fdc-a8f7-4031-9d69-27f2b1c35307-kube-api-access-xbpg2\" (UniqueName: \"kubernetes.io/projected/00581fdc-a8f7-4031-9d69-27f2b1c35307-kube-api-access-xbpg2\") pod \"00581fdc-a8f7-4031-9d69-27f2b1c35307\" (UID: \"00581fdc-a8f7-4031-9d69-27f2b1c35307\") " Oct 31 21:42:55.051207 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.051069 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00581fdc-a8f7-4031-9d69-27f2b1c35307-plugins-config-volume" pod "00581fdc-a8f7-4031-9d69-27f2b1c35307" (UID: "00581fdc-a8f7-4031-9d69-27f2b1c35307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:55.051387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.051211 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac7a720-aaab-4619-b3be-a0799a463093-plugins-config-volume" pod "8ac7a720-aaab-4619-b3be-a0799a463093" (UID: "8ac7a720-aaab-4619-b3be-a0799a463093"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:55.051466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.051431 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00581fdc-a8f7-4031-9d69-27f2b1c35307-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:55.051466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.051453 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ac7a720-aaab-4619-b3be-a0799a463093-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:55.055800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.055767 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00581fdc-a8f7-4031-9d69-27f2b1c35307-kube-api-access-xbpg2" pod "00581fdc-a8f7-4031-9d69-27f2b1c35307" (UID: "00581fdc-a8f7-4031-9d69-27f2b1c35307"). InnerVolumeSpecName "kube-api-access-xbpg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:55.056502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.056474 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac7a720-aaab-4619-b3be-a0799a463093-kube-api-access-pxzph" pod "8ac7a720-aaab-4619-b3be-a0799a463093" (UID: "8ac7a720-aaab-4619-b3be-a0799a463093"). InnerVolumeSpecName "kube-api-access-pxzph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:55.152404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.152360 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxzph\" (UniqueName: \"kubernetes.io/projected/8ac7a720-aaab-4619-b3be-a0799a463093-kube-api-access-pxzph\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:55.152404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.152400 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbpg2\" (UniqueName: \"kubernetes.io/projected/00581fdc-a8f7-4031-9d69-27f2b1c35307-kube-api-access-xbpg2\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:42:55.196860 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.196823 5188 generic.go:356] "Generic (PLEG): container finished" podID="00581fdc-a8f7-4031-9d69-27f2b1c35307" containerID="8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556" exitCode=0 Oct 31 21:42:55.197063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.196906 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" Oct 31 21:42:55.197063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.196910 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" event={"ID":"00581fdc-a8f7-4031-9d69-27f2b1c35307","Type":"ContainerDied","Data":"8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556"} Oct 31 21:42:55.197063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.197006 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8" event={"ID":"00581fdc-a8f7-4031-9d69-27f2b1c35307","Type":"ContainerDied","Data":"aff1d227a14a4b3a2ce97746b0cff6a88bd9eb6b4f0283a2e816754032ab863e"} Oct 31 21:42:55.197218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.197081 5188 scope.go:122] "RemoveContainer" containerID="8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556" Oct 31 21:42:55.201056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.201014 5188 generic.go:356] "Generic (PLEG): container finished" podID="8ac7a720-aaab-4619-b3be-a0799a463093" containerID="1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31" exitCode=0 Oct 31 21:42:55.201310 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.201085 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" event={"ID":"8ac7a720-aaab-4619-b3be-a0799a463093","Type":"ContainerDied","Data":"1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31"} Oct 31 21:42:55.201447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.201373 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" event={"ID":"8ac7a720-aaab-4619-b3be-a0799a463093","Type":"ContainerDied","Data":"8674c2cd7881c30f0d954fe4eb192109c882d055a906d4130b42ea758fb6da25"} Oct 31 21:42:55.201447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.201136 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4" Oct 31 21:42:55.210509 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.210486 5188 scope.go:122] "RemoveContainer" containerID="8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556" Oct 31 21:42:55.210834 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:55.210802 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556\": container with ID starting with 8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556 not found: ID does not exist" containerID="8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556" Oct 31 21:42:55.210959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.210842 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556"} err="failed to get container status \"8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556\": rpc error: code = NotFound desc = could not find container \"8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556\": container with ID starting with 8c2d6320b35f90990bda6da50c44f0900d42dd872380f1e21583cd64891f3556 not found: ID does not exist" Oct 31 21:42:55.210959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.210862 5188 scope.go:122] "RemoveContainer" containerID="1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31" Oct 31 21:42:55.224756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.224732 5188 scope.go:122] "RemoveContainer" containerID="1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31" Oct 31 21:42:55.225088 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:42:55.225064 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31\": container with ID starting with 1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31 not found: ID does not exist" containerID="1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31" Oct 31 21:42:55.225174 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.225096 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31"} err="failed to get container status \"1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31\": rpc error: code = NotFound desc = could not find container \"1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31\": container with ID starting with 1bdf81b9a6ecb3e6a1788148c8f5a66e8b45b336e8bf6516ccda00cca7e9fa31 not found: ID does not exist" Oct 31 21:42:55.257775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.257736 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8"] Oct 31 21:42:55.262397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.262365 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-774fc575ff-zvlk8"] Oct 31 21:42:55.291604 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.291559 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4"] Oct 31 21:42:55.300388 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.300350 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-694bf987bc-kq9j4"] Oct 31 21:42:55.476819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.476731 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00581fdc-a8f7-4031-9d69-27f2b1c35307" path="/var/lib/kubelet/pods/00581fdc-a8f7-4031-9d69-27f2b1c35307/volumes" Oct 31 21:42:55.477386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.477363 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac7a720-aaab-4619-b3be-a0799a463093" path="/var/lib/kubelet/pods/8ac7a720-aaab-4619-b3be-a0799a463093/volumes" Oct 31 21:42:55.830738 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.830620 5188 scope.go:122] "RemoveContainer" containerID="34f228c58be3eaa376b80c230963d4e0d4997027dde660c35ada35545bf607e7" Oct 31 21:42:55.843527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.843497 5188 scope.go:122] "RemoveContainer" containerID="83f09aa4d8a4454b4b4be7b38beb578a734ddea9bcffb09fe4803121580ac66c" Oct 31 21:42:55.858792 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.858764 5188 scope.go:122] "RemoveContainer" containerID="aba81d54be71901e2a1c8e9cfa82a4b7c80168c8daca64428cda8d378b27e056" Oct 31 21:42:55.915616 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:55.913503 5188 scope.go:122] "RemoveContainer" containerID="977e597e80f9f064032b08ed49147191224c8f30c0f085b2cf112ffe381f0b9f" Oct 31 21:42:56.124267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:42:56.124238 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:43:02.548141 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:43:02.548102 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:43:03.784248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.784208 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll"] Oct 31 21:43:03.785360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.785230 5188 cpu_manager.go:414] "containerMap: removing container" podUID="00581fdc-a8f7-4031-9d69-27f2b1c35307" containerName="payload-processing" Oct 31 21:43:03.785360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.785257 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00581fdc-a8f7-4031-9d69-27f2b1c35307" containerName="payload-processing" Oct 31 21:43:03.785360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.785282 5188 cpu_manager.go:414] "containerMap: removing container" podUID="8ac7a720-aaab-4619-b3be-a0799a463093" containerName="payload-pre-processing" Oct 31 21:43:03.785360 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.785290 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac7a720-aaab-4619-b3be-a0799a463093" containerName="payload-pre-processing" Oct 31 21:43:03.785667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.785540 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="00581fdc-a8f7-4031-9d69-27f2b1c35307" containerName="payload-processing" Oct 31 21:43:03.785667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.785559 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac7a720-aaab-4619-b3be-a0799a463093" containerName="payload-pre-processing" Oct 31 21:43:03.792594 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.792564 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.818648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.818524 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bb9990-serving-cert\"" Oct 31 21:43:03.823512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.823478 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll"] Oct 31 21:43:03.878989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.878952 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.878989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.878999 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-kube-api-access-2cbx8\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.879357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.879310 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-metrics-certs\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.879445 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.879399 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.981496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.981302 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.982129 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.982072 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.982260 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.982164 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-kube-api-access-2cbx8\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.982367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.982112 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.982891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.982429 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-metrics-certs\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.985711 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.985688 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-metrics-certs\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:03.985843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:03.985824 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:04.007983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:04.007054 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-kube-api-access-2cbx8\") pod \"maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:04.133239 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:04.133207 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:04.456605 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:04.456577 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll"] Oct 31 21:43:04.457020 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:43:04.456993 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd46ab77_7edd_408a_85b2_4bbce7bb4df0.slice/crio-734a3a281f868dc2ed8d9eba6411375c1f8ef95f2607a934ec5a5dc1ee0c935a WatchSource:0}: Error finding container 734a3a281f868dc2ed8d9eba6411375c1f8ef95f2607a934ec5a5dc1ee0c935a: Status 404 returned error can't find the container with id 734a3a281f868dc2ed8d9eba6411375c1f8ef95f2607a934ec5a5dc1ee0c935a Oct 31 21:43:05.321267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:05.321224 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" event={"ID":"dd46ab77-7edd-408a-85b2-4bbce7bb4df0","Type":"ContainerStarted","Data":"5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a"} Oct 31 21:43:05.321695 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:05.321315 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:05.321695 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:05.321348 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" event={"ID":"dd46ab77-7edd-408a-85b2-4bbce7bb4df0","Type":"ContainerStarted","Data":"734a3a281f868dc2ed8d9eba6411375c1f8ef95f2607a934ec5a5dc1ee0c935a"} Oct 31 21:43:05.360431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:05.360370 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" podStartSLOduration=2.05102818 podStartE2EDuration="2.360355175s" totalImagesPullingTime="309.326995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:05.359126822 +0000 UTC m=+1632.052084216" watchObservedRunningTime="2026-10-31 21:43:05.360355175 +0000 UTC m=+1632.053312617" Oct 31 21:43:11.335003 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.334965 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:43:11.557426 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.557387 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89"] Oct 31 21:43:11.563140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.563114 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:11.582087 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.582050 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f4569e\"" Oct 31 21:43:11.582245 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.582112 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f4569e-dockercfg-s5wqb\"" Oct 31 21:43:11.590593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.590504 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89"] Oct 31 21:43:11.606110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.605392 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6vr\" (UniqueName: \"kubernetes.io/projected/a4a78bb6-6094-454f-915a-a76064ab3b22-kube-api-access-gb6vr\") pod \"payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89\" (UID: \"a4a78bb6-6094-454f-915a-a76064ab3b22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:11.606110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.605619 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a78bb6-6094-454f-915a-a76064ab3b22-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89\" (UID: \"a4a78bb6-6094-454f-915a-a76064ab3b22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:11.676775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.676006 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226"] Oct 31 21:43:11.686513 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.686474 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:11.708878 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.708830 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdrq\" (UniqueName: \"kubernetes.io/projected/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-kube-api-access-zmdrq\") pod \"payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226\" (UID: \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:11.709078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.708924 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a78bb6-6094-454f-915a-a76064ab3b22-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89\" (UID: \"a4a78bb6-6094-454f-915a-a76064ab3b22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:11.709078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.709070 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6vr\" (UniqueName: \"kubernetes.io/projected/a4a78bb6-6094-454f-915a-a76064ab3b22-kube-api-access-gb6vr\") pod \"payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89\" (UID: \"a4a78bb6-6094-454f-915a-a76064ab3b22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:11.709186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.709160 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226\" (UID: \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:11.713987 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.713887 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a78bb6-6094-454f-915a-a76064ab3b22-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89\" (UID: \"a4a78bb6-6094-454f-915a-a76064ab3b22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:11.724972 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.724935 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226"] Oct 31 21:43:11.744391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.740070 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6vr\" (UniqueName: \"kubernetes.io/projected/a4a78bb6-6094-454f-915a-a76064ab3b22-kube-api-access-gb6vr\") pod \"payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89\" (UID: \"a4a78bb6-6094-454f-915a-a76064ab3b22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:11.809992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.809951 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226\" (UID: \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:11.810301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.810115 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdrq\" (UniqueName: \"kubernetes.io/projected/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-kube-api-access-zmdrq\") pod \"payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226\" (UID: \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:11.810716 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.810693 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226\" (UID: \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:11.832393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.832310 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdrq\" (UniqueName: \"kubernetes.io/projected/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-kube-api-access-zmdrq\") pod \"payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226\" (UID: \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:11.880288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:11.880247 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:12.027893 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:12.027577 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:12.208674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:12.208641 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89"] Oct 31 21:43:12.209393 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:43:12.209366 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a78bb6_6094_454f_915a_a76064ab3b22.slice/crio-569a1e96df7b64b58b6df5fd23d2d547865fa010d6c99fe72f3477784424d93b WatchSource:0}: Error finding container 569a1e96df7b64b58b6df5fd23d2d547865fa010d6c99fe72f3477784424d93b: Status 404 returned error can't find the container with id 569a1e96df7b64b58b6df5fd23d2d547865fa010d6c99fe72f3477784424d93b Oct 31 21:43:12.361487 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:43:12.361458 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode168f19b_bf27_4c7c_90a5_a9080d8dbc4d.slice/crio-f1c65926fbe958222a215a97781fe546d2ce2658e8f7d5f200902dd64e9b6bc3 WatchSource:0}: Error finding container f1c65926fbe958222a215a97781fe546d2ce2658e8f7d5f200902dd64e9b6bc3: Status 404 returned error can't find the container with id f1c65926fbe958222a215a97781fe546d2ce2658e8f7d5f200902dd64e9b6bc3 Oct 31 21:43:12.361839 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:12.361813 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226"] Oct 31 21:43:12.380293 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:12.380255 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" event={"ID":"a4a78bb6-6094-454f-915a-a76064ab3b22","Type":"ContainerStarted","Data":"406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256"} Oct 31 21:43:12.380480 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:12.380337 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" event={"ID":"a4a78bb6-6094-454f-915a-a76064ab3b22","Type":"ContainerStarted","Data":"569a1e96df7b64b58b6df5fd23d2d547865fa010d6c99fe72f3477784424d93b"} Oct 31 21:43:12.380592 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:12.380571 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:12.381929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:12.381829 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" event={"ID":"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d","Type":"ContainerStarted","Data":"f1c65926fbe958222a215a97781fe546d2ce2658e8f7d5f200902dd64e9b6bc3"} Oct 31 21:43:12.427384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:12.427311 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" podStartSLOduration=1.427296854 podStartE2EDuration="1.427296854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:12.425046886 +0000 UTC m=+1639.118004280" watchObservedRunningTime="2026-10-31 21:43:12.427296854 +0000 UTC m=+1639.120254246" Oct 31 21:43:13.249856 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.249815 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6"] Oct 31 21:43:13.255712 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.255682 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:13.282180 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.277636 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d43b66-dockercfg-jjmrl\"" Oct 31 21:43:13.282180 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.277770 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d43b66\"" Oct 31 21:43:13.289880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.286169 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6"] Oct 31 21:43:13.333182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.333144 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhqvw\" (UniqueName: \"kubernetes.io/projected/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-kube-api-access-qhqvw\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6\" (UID: \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:13.333391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.333205 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6\" (UID: \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:13.404872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.404821 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" event={"ID":"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d","Type":"ContainerStarted","Data":"b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f"} Oct 31 21:43:13.435493 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.435452 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhqvw\" (UniqueName: \"kubernetes.io/projected/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-kube-api-access-qhqvw\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6\" (UID: \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:13.435740 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.435507 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6\" (UID: \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:13.436530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.436497 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6\" (UID: \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:13.447766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.447727 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474"] Oct 31 21:43:13.452577 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.452555 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:13.458382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.458304 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" podStartSLOduration=2.458284135 podStartE2EDuration="2.458284135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:13.456860761 +0000 UTC m=+1640.149818156" watchObservedRunningTime="2026-10-31 21:43:13.458284135 +0000 UTC m=+1640.151241529" Oct 31 21:43:13.464292 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.464264 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhqvw\" (UniqueName: \"kubernetes.io/projected/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-kube-api-access-qhqvw\") pod \"payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6\" (UID: \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:13.483480 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.483442 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474"] Oct 31 21:43:13.536871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.536773 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494ded1a-7e61-490c-a73b-367af7969fe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474\" (UID: \"494ded1a-7e61-490c-a73b-367af7969fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:13.537070 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.537020 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khvb\" (UniqueName: \"kubernetes.io/projected/494ded1a-7e61-490c-a73b-367af7969fe0-kube-api-access-6khvb\") pod \"payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474\" (UID: \"494ded1a-7e61-490c-a73b-367af7969fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:13.583354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.580646 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:13.638427 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.638383 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khvb\" (UniqueName: \"kubernetes.io/projected/494ded1a-7e61-490c-a73b-367af7969fe0-kube-api-access-6khvb\") pod \"payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474\" (UID: \"494ded1a-7e61-490c-a73b-367af7969fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:13.638646 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.638496 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494ded1a-7e61-490c-a73b-367af7969fe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474\" (UID: \"494ded1a-7e61-490c-a73b-367af7969fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:13.639307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.639278 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494ded1a-7e61-490c-a73b-367af7969fe0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474\" (UID: \"494ded1a-7e61-490c-a73b-367af7969fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:13.700159 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.700087 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khvb\" (UniqueName: \"kubernetes.io/projected/494ded1a-7e61-490c-a73b-367af7969fe0-kube-api-access-6khvb\") pod \"payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474\" (UID: \"494ded1a-7e61-490c-a73b-367af7969fe0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:13.771203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.771164 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:13.915121 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:13.915078 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6"] Oct 31 21:43:13.916470 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:43:13.915373 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a257d42_2fe2_4027_b606_5dd0a86b1d2c.slice/crio-1ba48d318936c86288b5f1dfe519d061b9f000bc822d1610e302546e4e868349 WatchSource:0}: Error finding container 1ba48d318936c86288b5f1dfe519d061b9f000bc822d1610e302546e4e868349: Status 404 returned error can't find the container with id 1ba48d318936c86288b5f1dfe519d061b9f000bc822d1610e302546e4e868349 Oct 31 21:43:14.145840 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.145749 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474"] Oct 31 21:43:14.148751 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:43:14.148706 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494ded1a_7e61_490c_a73b_367af7969fe0.slice/crio-b4d6a070ab7c05b70c1e01ee94831f73e6bbb81fbf99c0b8dfb07ae2e9feda43 WatchSource:0}: Error finding container b4d6a070ab7c05b70c1e01ee94831f73e6bbb81fbf99c0b8dfb07ae2e9feda43: Status 404 returned error can't find the container with id b4d6a070ab7c05b70c1e01ee94831f73e6bbb81fbf99c0b8dfb07ae2e9feda43 Oct 31 21:43:14.418183 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.418077 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" event={"ID":"9a257d42-2fe2-4027-b606-5dd0a86b1d2c","Type":"ContainerStarted","Data":"1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0"} Oct 31 21:43:14.418793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.418771 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:14.418921 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.418906 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" event={"ID":"9a257d42-2fe2-4027-b606-5dd0a86b1d2c","Type":"ContainerStarted","Data":"1ba48d318936c86288b5f1dfe519d061b9f000bc822d1610e302546e4e868349"} Oct 31 21:43:14.421582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.420931 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" event={"ID":"494ded1a-7e61-490c-a73b-367af7969fe0","Type":"ContainerStarted","Data":"b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6"} Oct 31 21:43:14.421582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.420978 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" event={"ID":"494ded1a-7e61-490c-a73b-367af7969fe0","Type":"ContainerStarted","Data":"b4d6a070ab7c05b70c1e01ee94831f73e6bbb81fbf99c0b8dfb07ae2e9feda43"} Oct 31 21:43:14.421582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.421037 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:14.421582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.421151 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:14.474832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.474694 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" podStartSLOduration=1.474672091 podStartE2EDuration="1.474672091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:14.472826891 +0000 UTC m=+1641.165784287" watchObservedRunningTime="2026-10-31 21:43:14.474672091 +0000 UTC m=+1641.167629485" Oct 31 21:43:14.513494 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:14.513422 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" podStartSLOduration=1.5133975 podStartE2EDuration="1.5133975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:14.509757504 +0000 UTC m=+1641.202714900" watchObservedRunningTime="2026-10-31 21:43:14.5133975 +0000 UTC m=+1641.206354896" Oct 31 21:43:20.481907 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.481871 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp"] Oct 31 21:43:20.514687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.514648 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp"] Oct 31 21:43:20.514992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.514976 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:20.543767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.543730 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9c8b33\"" Oct 31 21:43:20.543927 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.543771 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9c8b33-dockercfg-948cl\"" Oct 31 21:43:20.638372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.638288 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460964d-80dd-41a2-89f1-3d5915432dd0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp\" (UID: \"5460964d-80dd-41a2-89f1-3d5915432dd0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:20.638581 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.638520 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhb2v\" (UniqueName: \"kubernetes.io/projected/5460964d-80dd-41a2-89f1-3d5915432dd0-kube-api-access-qhb2v\") pod \"payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp\" (UID: \"5460964d-80dd-41a2-89f1-3d5915432dd0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:20.740623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.740512 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460964d-80dd-41a2-89f1-3d5915432dd0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp\" (UID: \"5460964d-80dd-41a2-89f1-3d5915432dd0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:20.740623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.740562 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhb2v\" (UniqueName: \"kubernetes.io/projected/5460964d-80dd-41a2-89f1-3d5915432dd0-kube-api-access-qhb2v\") pod \"payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp\" (UID: \"5460964d-80dd-41a2-89f1-3d5915432dd0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:20.741568 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.741537 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460964d-80dd-41a2-89f1-3d5915432dd0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp\" (UID: \"5460964d-80dd-41a2-89f1-3d5915432dd0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:20.765955 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.765922 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhb2v\" (UniqueName: \"kubernetes.io/projected/5460964d-80dd-41a2-89f1-3d5915432dd0-kube-api-access-qhb2v\") pod \"payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp\" (UID: \"5460964d-80dd-41a2-89f1-3d5915432dd0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:20.840036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:20.839997 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:21.162182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:21.162145 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp"] Oct 31 21:43:21.162674 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:43:21.162648 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5460964d_80dd_41a2_89f1_3d5915432dd0.slice/crio-27e7da4ba87e8b9cce5d82e012e59d78296b712b64009084f74eee58394e3dcb WatchSource:0}: Error finding container 27e7da4ba87e8b9cce5d82e012e59d78296b712b64009084f74eee58394e3dcb: Status 404 returned error can't find the container with id 27e7da4ba87e8b9cce5d82e012e59d78296b712b64009084f74eee58394e3dcb Oct 31 21:43:21.525736 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:21.525647 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" event={"ID":"5460964d-80dd-41a2-89f1-3d5915432dd0","Type":"ContainerStarted","Data":"130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69"} Oct 31 21:43:21.525736 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:21.525702 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" event={"ID":"5460964d-80dd-41a2-89f1-3d5915432dd0","Type":"ContainerStarted","Data":"27e7da4ba87e8b9cce5d82e012e59d78296b712b64009084f74eee58394e3dcb"} Oct 31 21:43:21.526214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:21.525860 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:21.570763 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:21.570701 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" podStartSLOduration=1.570683531 podStartE2EDuration="1.570683531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:21.565751643 +0000 UTC m=+1648.258709037" watchObservedRunningTime="2026-10-31 21:43:21.570683531 +0000 UTC m=+1648.263640934" Oct 31 21:43:23.408123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:23.408088 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:43:25.431391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:25.431319 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:43:25.432497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:25.432477 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:43:26.438520 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:26.438487 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:43:32.537209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:32.537173 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:43:32.545031 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:43:32.544974 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:43:39.731447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:39.731406 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb"] Oct 31 21:43:39.731957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:39.731705 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" containerID="cri-o://a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34" gracePeriod=30 Oct 31 21:43:40.989848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:40.989793 5188 patch_prober.go:29] interesting pod/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:43:40.990344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:40.989902 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:43:43.720528 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:43.720492 5188 generic.go:356] "Generic (PLEG): container finished" podID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerID="fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57" exitCode=0 Oct 31 21:43:43.721023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:43.720547 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" event={"ID":"58ba141e-72e4-4a29-be5c-fa7c9dae6317","Type":"ContainerDied","Data":"fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57"} Oct 31 21:43:43.721089 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:43.721047 5188 scope.go:122] "RemoveContainer" containerID="fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57" Oct 31 21:43:43.722863 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:43.722834 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:43:43.722941 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:43.722919 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:43:43.722983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:43.722953 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:43:44.010534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:44.010087 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:43:44.010534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:44.010175 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:43:44.730412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:44.730375 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" event={"ID":"58ba141e-72e4-4a29-be5c-fa7c9dae6317","Type":"ContainerStarted","Data":"12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad"} Oct 31 21:43:45.123867 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.123829 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:43:45.220372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.213923 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:43:45.220372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.214102 5188 patch_prober.go:29] interesting pod/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 31 21:43:45.220372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.214152 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 31 21:43:45.237870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.237568 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-socket\") pod \"5ca228a8-ae18-4f21-8238-a23e53143a5e\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " Oct 31 21:43:45.237870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.237679 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-podinfo\") pod \"5ca228a8-ae18-4f21-8238-a23e53143a5e\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " Oct 31 21:43:45.237870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.237736 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-credential-socket\") pod \"5ca228a8-ae18-4f21-8238-a23e53143a5e\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " Oct 31 21:43:45.237870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.237778 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-kube-api-access-hthc4\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-kube-api-access-hthc4\") pod \"5ca228a8-ae18-4f21-8238-a23e53143a5e\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " Oct 31 21:43:45.237870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.237865 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca228a8-ae18-4f21-8238-a23e53143a5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca228a8-ae18-4f21-8238-a23e53143a5e-istiod-ca-cert\") pod \"5ca228a8-ae18-4f21-8238-a23e53143a5e\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " Oct 31 21:43:45.238339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.237898 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-token\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-token\") pod \"5ca228a8-ae18-4f21-8238-a23e53143a5e\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " Oct 31 21:43:45.238339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.237978 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-certs\") pod \"5ca228a8-ae18-4f21-8238-a23e53143a5e\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " Oct 31 21:43:45.238339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.238014 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-envoy\") pod \"5ca228a8-ae18-4f21-8238-a23e53143a5e\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " Oct 31 21:43:45.238339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.238043 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-data\") pod \"5ca228a8-ae18-4f21-8238-a23e53143a5e\" (UID: \"5ca228a8-ae18-4f21-8238-a23e53143a5e\") " Oct 31 21:43:45.240256 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.240217 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-certs" pod "5ca228a8-ae18-4f21-8238-a23e53143a5e" (UID: "5ca228a8-ae18-4f21-8238-a23e53143a5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:45.240586 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.240559 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-data" pod "5ca228a8-ae18-4f21-8238-a23e53143a5e" (UID: "5ca228a8-ae18-4f21-8238-a23e53143a5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:45.241481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.241449 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-credential-socket" pod "5ca228a8-ae18-4f21-8238-a23e53143a5e" (UID: "5ca228a8-ae18-4f21-8238-a23e53143a5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:45.241686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.241652 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca228a8-ae18-4f21-8238-a23e53143a5e-istiod-ca-cert" pod "5ca228a8-ae18-4f21-8238-a23e53143a5e" (UID: "5ca228a8-ae18-4f21-8238-a23e53143a5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:45.241765 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.241747 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-socket" pod "5ca228a8-ae18-4f21-8238-a23e53143a5e" (UID: "5ca228a8-ae18-4f21-8238-a23e53143a5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:45.243460 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.243425 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-token" pod "5ca228a8-ae18-4f21-8238-a23e53143a5e" (UID: "5ca228a8-ae18-4f21-8238-a23e53143a5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:45.244396 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.244367 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-kube-api-access-hthc4" pod "5ca228a8-ae18-4f21-8238-a23e53143a5e" (UID: "5ca228a8-ae18-4f21-8238-a23e53143a5e"). InnerVolumeSpecName "kube-api-access-hthc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:45.247012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.246975 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-envoy" pod "5ca228a8-ae18-4f21-8238-a23e53143a5e" (UID: "5ca228a8-ae18-4f21-8238-a23e53143a5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:45.249182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.249151 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-podinfo" pod "5ca228a8-ae18-4f21-8238-a23e53143a5e" (UID: "5ca228a8-ae18-4f21-8238-a23e53143a5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:43:45.340438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.340344 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:45.340438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.340389 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:45.340438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.340403 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hthc4\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-kube-api-access-hthc4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:45.340438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.340414 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ca228a8-ae18-4f21-8238-a23e53143a5e-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:45.340438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.340423 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:45.340438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.340432 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:45.340438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.340439 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:45.340438 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.340447 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:45.340851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.340454 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ca228a8-ae18-4f21-8238-a23e53143a5e-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:45.741216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.741178 5188 generic.go:356] "Generic (PLEG): container finished" podID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerID="a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34" exitCode=0 Oct 31 21:43:45.741814 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.741261 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" Oct 31 21:43:45.741814 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.741727 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" event={"ID":"5ca228a8-ae18-4f21-8238-a23e53143a5e","Type":"ContainerDied","Data":"a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34"} Oct 31 21:43:45.741814 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.741791 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb" event={"ID":"5ca228a8-ae18-4f21-8238-a23e53143a5e","Type":"ContainerDied","Data":"d5775470930f17aecf876df003938f4a95f3421e3f1a5fa60840dbfaa70f5305"} Oct 31 21:43:45.741989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.741821 5188 scope.go:122] "RemoveContainer" containerID="a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34" Oct 31 21:43:45.755134 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.755102 5188 scope.go:122] "RemoveContainer" containerID="0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826" Oct 31 21:43:45.766467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.766403 5188 scope.go:122] "RemoveContainer" containerID="a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34" Oct 31 21:43:45.766770 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:43:45.766752 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34\": container with ID starting with a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34 not found: ID does not exist" containerID="a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34" Oct 31 21:43:45.766817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.766782 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34"} err="failed to get container status \"a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34\": rpc error: code = NotFound desc = could not find container \"a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34\": container with ID starting with a3414f46e157a15d32b91cc1a0418afa386fa9efef463a4682906657f9695e34 not found: ID does not exist" Oct 31 21:43:45.766817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.766804 5188 scope.go:122] "RemoveContainer" containerID="0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826" Oct 31 21:43:45.767124 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:43:45.767099 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826\": container with ID starting with 0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826 not found: ID does not exist" containerID="0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826" Oct 31 21:43:45.767242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.767127 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826"} err="failed to get container status \"0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826\": rpc error: code = NotFound desc = could not find container \"0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826\": container with ID starting with 0523f3f9adafeca47319acc9e6636b5a7217bcd0e86787dad26ff394badfb826 not found: ID does not exist" Oct 31 21:43:45.806550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.806513 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb"] Oct 31 21:43:45.812949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:45.812907 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-x2llb"] Oct 31 21:43:46.198503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:46.198462 5188 patch_prober.go:29] interesting pod/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 31 21:43:46.198820 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:46.198789 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 31 21:43:47.195175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:47.195136 5188 patch_prober.go:29] interesting pod/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 31 21:43:47.195824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:47.195789 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 31 21:43:47.498409 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:47.497712 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" path="/var/lib/kubelet/pods/5ca228a8-ae18-4f21-8238-a23e53143a5e/volumes" Oct 31 21:43:48.199380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:48.199311 5188 patch_prober.go:29] interesting pod/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 31 21:43:48.200022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:48.199986 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 31 21:43:49.199092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.199049 5188 patch_prober.go:29] interesting pod/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 31 21:43:49.199432 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.199398 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 31 21:43:49.219798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.219758 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf"] Oct 31 21:43:49.220866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.220834 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" Oct 31 21:43:49.220866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.220863 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" Oct 31 21:43:49.221037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.220904 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" Oct 31 21:43:49.221037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.220910 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" Oct 31 21:43:49.221106 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.221040 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" Oct 31 21:43:49.221106 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.221053 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca228a8-ae18-4f21-8238-a23e53143a5e" containerName="istio-proxy" Oct 31 21:43:49.224787 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.224766 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.240041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.240013 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-9c8b33-serving-cert\"" Oct 31 21:43:49.248367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.245523 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf"] Oct 31 21:43:49.325428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.325277 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkk8v\" (UniqueName: \"kubernetes.io/projected/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-kube-api-access-bkk8v\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.325428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.325418 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-metrics-certs\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.325699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.325493 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.325699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.325532 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.426782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.426740 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkk8v\" (UniqueName: \"kubernetes.io/projected/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-kube-api-access-bkk8v\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.426992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.426860 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-metrics-certs\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.426992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.426931 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.426992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.426966 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.427821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.427756 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.429560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.429537 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-maas-api-tls\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.429658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.429582 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-metrics-certs\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.448823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.448770 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkk8v\" (UniqueName: \"kubernetes.io/projected/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-kube-api-access-bkk8v\") pod \"maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.539941 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.539839 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:49.912998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:49.912191 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf"] Oct 31 21:43:50.194686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:50.194574 5188 patch_prober.go:29] interesting pod/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 31 21:43:50.194686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:50.194644 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 31 21:43:50.827304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:50.827200 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" event={"ID":"752aebb7-9f3c-4c16-aea5-e08ff2a9d312","Type":"ContainerStarted","Data":"b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8"} Oct 31 21:43:50.827304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:50.827289 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" event={"ID":"752aebb7-9f3c-4c16-aea5-e08ff2a9d312","Type":"ContainerStarted","Data":"02b7147231f694baa6ff018d609ed1e1ec34a51a788fdf15aba6faf2e2a2e2f0"} Oct 31 21:43:50.828014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:50.827372 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:51.199981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:51.199951 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:43:51.200459 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:51.200435 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:43:51.201147 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:51.201126 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:43:51.255590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:51.255525 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" podStartSLOduration=1.815787176 podStartE2EDuration="2.255510171s" totalImagesPullingTime="439.722995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:50.889631576 +0000 UTC m=+1677.582588969" watchObservedRunningTime="2026-10-31 21:43:51.255510171 +0000 UTC m=+1677.948467562" Oct 31 21:43:53.355507 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:53.355471 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52"] Oct 31 21:43:53.383923 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:53.383886 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq"] Oct 31 21:43:54.427757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.427719 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl"] Oct 31 21:43:54.428254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.428015 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" podUID="38841d38-152d-4315-a8b8-7eba09d98ebf" containerName="maas-api" containerID="cri-o://dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7" gracePeriod=30 Oct 31 21:43:54.789168 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.789126 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:43:54.829479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.829440 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-maas-api-tls\") pod \"38841d38-152d-4315-a8b8-7eba09d98ebf\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " Oct 31 21:43:54.829674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.829550 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38841d38-152d-4315-a8b8-7eba09d98ebf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38841d38-152d-4315-a8b8-7eba09d98ebf-openshift-service-ca\") pod \"38841d38-152d-4315-a8b8-7eba09d98ebf\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " Oct 31 21:43:54.829674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.829573 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-metrics-certs\") pod \"38841d38-152d-4315-a8b8-7eba09d98ebf\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " Oct 31 21:43:54.829796 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.829710 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38841d38-152d-4315-a8b8-7eba09d98ebf-kube-api-access-xw6tn\" (UniqueName: \"kubernetes.io/projected/38841d38-152d-4315-a8b8-7eba09d98ebf-kube-api-access-xw6tn\") pod \"38841d38-152d-4315-a8b8-7eba09d98ebf\" (UID: \"38841d38-152d-4315-a8b8-7eba09d98ebf\") " Oct 31 21:43:54.829963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.829928 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38841d38-152d-4315-a8b8-7eba09d98ebf-openshift-service-ca" pod "38841d38-152d-4315-a8b8-7eba09d98ebf" (UID: "38841d38-152d-4315-a8b8-7eba09d98ebf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:54.830349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.830300 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38841d38-152d-4315-a8b8-7eba09d98ebf-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:54.832048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.832023 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-maas-api-tls" pod "38841d38-152d-4315-a8b8-7eba09d98ebf" (UID: "38841d38-152d-4315-a8b8-7eba09d98ebf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:43:54.842257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.842207 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38841d38-152d-4315-a8b8-7eba09d98ebf-kube-api-access-xw6tn" pod "38841d38-152d-4315-a8b8-7eba09d98ebf" (UID: "38841d38-152d-4315-a8b8-7eba09d98ebf"). InnerVolumeSpecName "kube-api-access-xw6tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:54.851499 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.851462 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-metrics-certs" pod "38841d38-152d-4315-a8b8-7eba09d98ebf" (UID: "38841d38-152d-4315-a8b8-7eba09d98ebf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:43:54.935937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.935900 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw6tn\" (UniqueName: \"kubernetes.io/projected/38841d38-152d-4315-a8b8-7eba09d98ebf-kube-api-access-xw6tn\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:54.935937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.935939 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:54.936232 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.935953 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38841d38-152d-4315-a8b8-7eba09d98ebf-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:54.966596 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.966540 5188 generic.go:356] "Generic (PLEG): container finished" podID="38841d38-152d-4315-a8b8-7eba09d98ebf" containerID="dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7" exitCode=0 Oct 31 21:43:54.973390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.967092 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" event={"ID":"38841d38-152d-4315-a8b8-7eba09d98ebf","Type":"ContainerDied","Data":"dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7"} Oct 31 21:43:54.973390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.967165 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" event={"ID":"38841d38-152d-4315-a8b8-7eba09d98ebf","Type":"ContainerDied","Data":"b36b84fc2b8fd889321a16e7611c6dfbf2246175163955fd5c31b79a79024441"} Oct 31 21:43:54.973390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.967199 5188 scope.go:122] "RemoveContainer" containerID="dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7" Oct 31 21:43:54.973390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.967384 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl" Oct 31 21:43:54.999767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:54.999746 5188 scope.go:122] "RemoveContainer" containerID="dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7" Oct 31 21:43:55.000042 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:43:55.000017 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7\": container with ID starting with dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7 not found: ID does not exist" containerID="dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7" Oct 31 21:43:55.000114 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:55.000052 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7"} err="failed to get container status \"dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7\": rpc error: code = NotFound desc = could not find container \"dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7\": container with ID starting with dd5f9126aa461a61a80438ebf2071aad59e676d031a009f606f37acb1fc7bfa7 not found: ID does not exist" Oct 31 21:43:55.032525 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:55.032488 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl"] Oct 31 21:43:55.039730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:55.039700 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-6d6576-6879ffcbd7-ffkwl"] Oct 31 21:43:55.477255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:55.477211 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38841d38-152d-4315-a8b8-7eba09d98ebf" path="/var/lib/kubelet/pods/38841d38-152d-4315-a8b8-7eba09d98ebf/volumes" Oct 31 21:43:56.065753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:56.065719 5188 scope.go:122] "RemoveContainer" containerID="cd8cbd44fe059ac5743b0c61c758a73d286e226c84d1ffaf405f3b0b36bac325" Oct 31 21:43:56.841059 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:56.841024 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:43:58.356460 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.356386 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" podUID="82289a34-8b1f-42cf-b716-f21732bcf54f" containerName="payload-processing" containerID="cri-o://bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9" gracePeriod=25 Oct 31 21:43:58.384661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.384613 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" podUID="962b779d-da15-4513-ade5-b5a289b411dd" containerName="payload-pre-processing" containerID="cri-o://bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c" gracePeriod=25 Oct 31 21:43:58.457302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.457261 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn"] Oct 31 21:43:58.458059 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.457606 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" podUID="7a00a2fc-729d-4fcb-961b-812f90b42430" containerName="maas-api" containerID="cri-o://a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803" gracePeriod=30 Oct 31 21:43:58.682963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.681396 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8"] Oct 31 21:43:58.682963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.681677 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" podUID="7f20f387-c81d-41eb-804b-6b992484f56b" containerName="payload-processing" containerID="cri-o://7099ce4a6fe453355fe0f96027c20fbf6cbf2e703d2735723305abccac9634b5" gracePeriod=30 Oct 31 21:43:58.689058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.688928 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb"] Oct 31 21:43:58.689209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.689179 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" podUID="82aa5bfd-7b7d-435f-a76f-36c14c0f0251" containerName="payload-pre-processing" containerID="cri-o://dd2041ca1cb1f5dd1233e3955931c5cdab6222ecf9ccf699738ca49f8ce1da6a" gracePeriod=30 Oct 31 21:43:58.815078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.811604 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:43:58.846209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.846177 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:43:58.888721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.887510 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:43:58.990302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.928223 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962b779d-da15-4513-ade5-b5a289b411dd-kube-api-access-225gk\" (UniqueName: \"kubernetes.io/projected/962b779d-da15-4513-ade5-b5a289b411dd-kube-api-access-225gk\") pod \"962b779d-da15-4513-ade5-b5a289b411dd\" (UID: \"962b779d-da15-4513-ade5-b5a289b411dd\") " Oct 31 21:43:58.990302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.928291 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/962b779d-da15-4513-ade5-b5a289b411dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962b779d-da15-4513-ade5-b5a289b411dd-plugins-config-volume\") pod \"962b779d-da15-4513-ade5-b5a289b411dd\" (UID: \"962b779d-da15-4513-ade5-b5a289b411dd\") " Oct 31 21:43:58.993949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.992727 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-metrics-certs\") pod \"7a00a2fc-729d-4fcb-961b-812f90b42430\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " Oct 31 21:43:58.993949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.992798 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82289a34-8b1f-42cf-b716-f21732bcf54f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82289a34-8b1f-42cf-b716-f21732bcf54f-plugins-config-volume\") pod \"82289a34-8b1f-42cf-b716-f21732bcf54f\" (UID: \"82289a34-8b1f-42cf-b716-f21732bcf54f\") " Oct 31 21:43:58.993949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.992853 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82289a34-8b1f-42cf-b716-f21732bcf54f-kube-api-access-lzxx5\" (UniqueName: \"kubernetes.io/projected/82289a34-8b1f-42cf-b716-f21732bcf54f-kube-api-access-lzxx5\") pod \"82289a34-8b1f-42cf-b716-f21732bcf54f\" (UID: \"82289a34-8b1f-42cf-b716-f21732bcf54f\") " Oct 31 21:43:58.993949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.993006 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a00a2fc-729d-4fcb-961b-812f90b42430-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a00a2fc-729d-4fcb-961b-812f90b42430-openshift-service-ca\") pod \"7a00a2fc-729d-4fcb-961b-812f90b42430\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " Oct 31 21:43:58.994943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.994511 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a00a2fc-729d-4fcb-961b-812f90b42430-openshift-service-ca" pod "7a00a2fc-729d-4fcb-961b-812f90b42430" (UID: "7a00a2fc-729d-4fcb-961b-812f90b42430"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:58.994943 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:58.994680 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82289a34-8b1f-42cf-b716-f21732bcf54f-plugins-config-volume" pod "82289a34-8b1f-42cf-b716-f21732bcf54f" (UID: "82289a34-8b1f-42cf-b716-f21732bcf54f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:59.053794 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.053750 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962b779d-da15-4513-ade5-b5a289b411dd-plugins-config-volume" pod "962b779d-da15-4513-ade5-b5a289b411dd" (UID: "962b779d-da15-4513-ade5-b5a289b411dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:59.068610 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.068204 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-metrics-certs" pod "7a00a2fc-729d-4fcb-961b-812f90b42430" (UID: "7a00a2fc-729d-4fcb-961b-812f90b42430"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:43:59.068610 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.068569 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82289a34-8b1f-42cf-b716-f21732bcf54f-kube-api-access-lzxx5" pod "82289a34-8b1f-42cf-b716-f21732bcf54f" (UID: "82289a34-8b1f-42cf-b716-f21732bcf54f"). InnerVolumeSpecName "kube-api-access-lzxx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:59.075380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.072517 5188 generic.go:356] "Generic (PLEG): container finished" podID="82289a34-8b1f-42cf-b716-f21732bcf54f" containerID="bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9" exitCode=0 Oct 31 21:43:59.075380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.072637 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" event={"ID":"82289a34-8b1f-42cf-b716-f21732bcf54f","Type":"ContainerDied","Data":"bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9"} Oct 31 21:43:59.075380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.072692 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" event={"ID":"82289a34-8b1f-42cf-b716-f21732bcf54f","Type":"ContainerDied","Data":"405610b5de84b590e5e2d5e49bbdb96149a27fbfcd4f1376e1b73c8e11878b95"} Oct 31 21:43:59.075380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.072720 5188 scope.go:122] "RemoveContainer" containerID="bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9" Oct 31 21:43:59.075380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.072891 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52" Oct 31 21:43:59.097358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.095593 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-maas-api-tls\") pod \"7a00a2fc-729d-4fcb-961b-812f90b42430\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " Oct 31 21:43:59.097358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.096015 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a00a2fc-729d-4fcb-961b-812f90b42430-kube-api-access-ktvjv\" (UniqueName: \"kubernetes.io/projected/7a00a2fc-729d-4fcb-961b-812f90b42430-kube-api-access-ktvjv\") pod \"7a00a2fc-729d-4fcb-961b-812f90b42430\" (UID: \"7a00a2fc-729d-4fcb-961b-812f90b42430\") " Oct 31 21:43:59.098064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.097691 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.098064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.097728 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82289a34-8b1f-42cf-b716-f21732bcf54f-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.098064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.097745 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzxx5\" (UniqueName: \"kubernetes.io/projected/82289a34-8b1f-42cf-b716-f21732bcf54f-kube-api-access-lzxx5\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.098064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.097759 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a00a2fc-729d-4fcb-961b-812f90b42430-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.098064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.097774 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962b779d-da15-4513-ade5-b5a289b411dd-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.099422 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.099090 5188 generic.go:356] "Generic (PLEG): container finished" podID="7f20f387-c81d-41eb-804b-6b992484f56b" containerID="7099ce4a6fe453355fe0f96027c20fbf6cbf2e703d2735723305abccac9634b5" exitCode=0 Oct 31 21:43:59.099422 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.099244 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" event={"ID":"7f20f387-c81d-41eb-804b-6b992484f56b","Type":"ContainerDied","Data":"7099ce4a6fe453355fe0f96027c20fbf6cbf2e703d2735723305abccac9634b5"} Oct 31 21:43:59.104824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.104508 5188 generic.go:356] "Generic (PLEG): container finished" podID="82aa5bfd-7b7d-435f-a76f-36c14c0f0251" containerID="dd2041ca1cb1f5dd1233e3955931c5cdab6222ecf9ccf699738ca49f8ce1da6a" exitCode=0 Oct 31 21:43:59.104824 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.104715 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" event={"ID":"82aa5bfd-7b7d-435f-a76f-36c14c0f0251","Type":"ContainerDied","Data":"dd2041ca1cb1f5dd1233e3955931c5cdab6222ecf9ccf699738ca49f8ce1da6a"} Oct 31 21:43:59.108884 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.108245 5188 generic.go:356] "Generic (PLEG): container finished" podID="7a00a2fc-729d-4fcb-961b-812f90b42430" containerID="a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803" exitCode=0 Oct 31 21:43:59.108884 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.108387 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" event={"ID":"7a00a2fc-729d-4fcb-961b-812f90b42430","Type":"ContainerDied","Data":"a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803"} Oct 31 21:43:59.108884 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.108457 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" event={"ID":"7a00a2fc-729d-4fcb-961b-812f90b42430","Type":"ContainerDied","Data":"7aabaeed605237feff5d35f73c8e8c9381d4450b407e9576f9a3e7e02cd696f5"} Oct 31 21:43:59.108884 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.108558 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn" Oct 31 21:43:59.112437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.111864 5188 generic.go:356] "Generic (PLEG): container finished" podID="962b779d-da15-4513-ade5-b5a289b411dd" containerID="bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c" exitCode=0 Oct 31 21:43:59.112437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.111961 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" event={"ID":"962b779d-da15-4513-ade5-b5a289b411dd","Type":"ContainerDied","Data":"bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c"} Oct 31 21:43:59.112437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.112008 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" event={"ID":"962b779d-da15-4513-ade5-b5a289b411dd","Type":"ContainerDied","Data":"b28ba4e020ecf8dc0448731c141b2790f7bc2a9c3651c6cb8d8473bfc08421ee"} Oct 31 21:43:59.112437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.112083 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq" Oct 31 21:43:59.112968 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.112914 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a00a2fc-729d-4fcb-961b-812f90b42430-kube-api-access-ktvjv" pod "7a00a2fc-729d-4fcb-961b-812f90b42430" (UID: "7a00a2fc-729d-4fcb-961b-812f90b42430"). InnerVolumeSpecName "kube-api-access-ktvjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:59.113668 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.113597 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-maas-api-tls" pod "7a00a2fc-729d-4fcb-961b-812f90b42430" (UID: "7a00a2fc-729d-4fcb-961b-812f90b42430"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:43:59.140271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.138545 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962b779d-da15-4513-ade5-b5a289b411dd-kube-api-access-225gk" pod "962b779d-da15-4513-ade5-b5a289b411dd" (UID: "962b779d-da15-4513-ade5-b5a289b411dd"). InnerVolumeSpecName "kube-api-access-225gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:59.146485 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.146448 5188 scope.go:122] "RemoveContainer" containerID="bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9" Oct 31 21:43:59.146940 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:43:59.146905 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9\": container with ID starting with bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9 not found: ID does not exist" containerID="bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9" Oct 31 21:43:59.147055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.146955 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9"} err="failed to get container status \"bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9\": rpc error: code = NotFound desc = could not find container \"bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9\": container with ID starting with bfdf52fdfcaf08aecc66d7580058377b4bde9fb00320f055860cfb4a5cd2f1b9 not found: ID does not exist" Oct 31 21:43:59.147055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.146983 5188 scope.go:122] "RemoveContainer" containerID="a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803" Oct 31 21:43:59.148482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.148457 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52"] Oct 31 21:43:59.154847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.154811 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6d6576-7b97b7b6c-txn52"] Oct 31 21:43:59.164550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.164515 5188 scope.go:122] "RemoveContainer" containerID="a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803" Oct 31 21:43:59.165233 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:43:59.165196 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803\": container with ID starting with a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803 not found: ID does not exist" containerID="a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803" Oct 31 21:43:59.165387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.165256 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803"} err="failed to get container status \"a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803\": rpc error: code = NotFound desc = could not find container \"a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803\": container with ID starting with a2eea357d85d72d046a4fc60eec81fecf2fb44f7d9edd7da5407ea3ea6f86803 not found: ID does not exist" Oct 31 21:43:59.165387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.165283 5188 scope.go:122] "RemoveContainer" containerID="bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c" Oct 31 21:43:59.190724 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.188632 5188 scope.go:122] "RemoveContainer" containerID="bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c" Oct 31 21:43:59.190724 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:43:59.189085 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c\": container with ID starting with bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c not found: ID does not exist" containerID="bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c" Oct 31 21:43:59.190724 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.189129 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c"} err="failed to get container status \"bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c\": rpc error: code = NotFound desc = could not find container \"bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c\": container with ID starting with bc884bb7731050d7efc28d06e827fe0d29ab1b2e07814149a44a9da7e8c3ff6c not found: ID does not exist" Oct 31 21:43:59.199917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.198981 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-225gk\" (UniqueName: \"kubernetes.io/projected/962b779d-da15-4513-ade5-b5a289b411dd-kube-api-access-225gk\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.199917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.199020 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktvjv\" (UniqueName: \"kubernetes.io/projected/7a00a2fc-729d-4fcb-961b-812f90b42430-kube-api-access-ktvjv\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.199917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.199037 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a00a2fc-729d-4fcb-961b-812f90b42430-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.237370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.237338 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:43:59.300651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.300545 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f20f387-c81d-41eb-804b-6b992484f56b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f20f387-c81d-41eb-804b-6b992484f56b-plugins-config-volume\") pod \"7f20f387-c81d-41eb-804b-6b992484f56b\" (UID: \"7f20f387-c81d-41eb-804b-6b992484f56b\") " Oct 31 21:43:59.300838 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.300665 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f20f387-c81d-41eb-804b-6b992484f56b-kube-api-access-pp245\" (UniqueName: \"kubernetes.io/projected/7f20f387-c81d-41eb-804b-6b992484f56b-kube-api-access-pp245\") pod \"7f20f387-c81d-41eb-804b-6b992484f56b\" (UID: \"7f20f387-c81d-41eb-804b-6b992484f56b\") " Oct 31 21:43:59.301388 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.301309 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f20f387-c81d-41eb-804b-6b992484f56b-plugins-config-volume" pod "7f20f387-c81d-41eb-804b-6b992484f56b" (UID: "7f20f387-c81d-41eb-804b-6b992484f56b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:59.304158 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.304100 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f20f387-c81d-41eb-804b-6b992484f56b-kube-api-access-pp245" pod "7f20f387-c81d-41eb-804b-6b992484f56b" (UID: "7f20f387-c81d-41eb-804b-6b992484f56b"). InnerVolumeSpecName "kube-api-access-pp245". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:59.317874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.317834 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:43:59.402632 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.402596 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-plugins-config-volume\") pod \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\" (UID: \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\") " Oct 31 21:43:59.403248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.402719 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-kube-api-access-jgsbp\" (UniqueName: \"kubernetes.io/projected/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-kube-api-access-jgsbp\") pod \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\" (UID: \"82aa5bfd-7b7d-435f-a76f-36c14c0f0251\") " Oct 31 21:43:59.403248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.403133 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-plugins-config-volume" pod "82aa5bfd-7b7d-435f-a76f-36c14c0f0251" (UID: "82aa5bfd-7b7d-435f-a76f-36c14c0f0251"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:59.403908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.403873 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp245\" (UniqueName: \"kubernetes.io/projected/7f20f387-c81d-41eb-804b-6b992484f56b-kube-api-access-pp245\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.403908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.403905 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.404205 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.403921 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f20f387-c81d-41eb-804b-6b992484f56b-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.416139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.416094 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-kube-api-access-jgsbp" pod "82aa5bfd-7b7d-435f-a76f-36c14c0f0251" (UID: "82aa5bfd-7b7d-435f-a76f-36c14c0f0251"). InnerVolumeSpecName "kube-api-access-jgsbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:59.477889 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.477849 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82289a34-8b1f-42cf-b716-f21732bcf54f" path="/var/lib/kubelet/pods/82289a34-8b1f-42cf-b716-f21732bcf54f/volumes" Oct 31 21:43:59.480418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.480384 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn"] Oct 31 21:43:59.483284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.483234 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d75c94-5598569766-xl8sn"] Oct 31 21:43:59.511687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.511648 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgsbp\" (UniqueName: \"kubernetes.io/projected/82aa5bfd-7b7d-435f-a76f-36c14c0f0251-kube-api-access-jgsbp\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:43:59.524951 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.524908 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq"] Oct 31 21:43:59.529521 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:43:59.529488 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6d6576-f4dc846ff-dg6gq"] Oct 31 21:44:00.125036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.125002 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" Oct 31 21:44:00.125036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.125024 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8" event={"ID":"7f20f387-c81d-41eb-804b-6b992484f56b","Type":"ContainerDied","Data":"d7f90fc1749fa53e53251b73e0c53c3726f4bb83de174514541eb480b4c420c6"} Oct 31 21:44:00.125311 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.125122 5188 scope.go:122] "RemoveContainer" containerID="7099ce4a6fe453355fe0f96027c20fbf6cbf2e703d2735723305abccac9634b5" Oct 31 21:44:00.126514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.126487 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" Oct 31 21:44:00.126590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.126484 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb" event={"ID":"82aa5bfd-7b7d-435f-a76f-36c14c0f0251","Type":"ContainerDied","Data":"04eff68c149a4e7eb6612cf867444c249b356b75bbb22f728dfd4aab60dad88b"} Oct 31 21:44:00.137240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.137216 5188 scope.go:122] "RemoveContainer" containerID="dd2041ca1cb1f5dd1233e3955931c5cdab6222ecf9ccf699738ca49f8ce1da6a" Oct 31 21:44:00.175623 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.175596 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8"] Oct 31 21:44:00.188772 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.188743 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d75c94-655fd7c9cb-4tcp8"] Oct 31 21:44:00.215768 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.215731 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb"] Oct 31 21:44:00.220957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:00.220921 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d75c94-5dc8c9458c-ptvmb"] Oct 31 21:44:01.477901 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:01.477868 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a00a2fc-729d-4fcb-961b-812f90b42430" path="/var/lib/kubelet/pods/7a00a2fc-729d-4fcb-961b-812f90b42430/volumes" Oct 31 21:44:01.479062 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:01.479031 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f20f387-c81d-41eb-804b-6b992484f56b" path="/var/lib/kubelet/pods/7f20f387-c81d-41eb-804b-6b992484f56b/volumes" Oct 31 21:44:01.479676 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:01.479651 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82aa5bfd-7b7d-435f-a76f-36c14c0f0251" path="/var/lib/kubelet/pods/82aa5bfd-7b7d-435f-a76f-36c14c0f0251/volumes" Oct 31 21:44:01.480657 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:01.480632 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962b779d-da15-4513-ade5-b5a289b411dd" path="/var/lib/kubelet/pods/962b779d-da15-4513-ade5-b5a289b411dd/volumes" Oct 31 21:44:02.538622 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:44:02.538587 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:44:04.810571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.810530 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn"] Oct 31 21:44:04.811350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811283 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7a00a2fc-729d-4fcb-961b-812f90b42430" containerName="maas-api" Oct 31 21:44:04.811350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811307 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a00a2fc-729d-4fcb-961b-812f90b42430" containerName="maas-api" Oct 31 21:44:04.811350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811343 5188 cpu_manager.go:414] "containerMap: removing container" podUID="82289a34-8b1f-42cf-b716-f21732bcf54f" containerName="payload-processing" Oct 31 21:44:04.811350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811351 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82289a34-8b1f-42cf-b716-f21732bcf54f" containerName="payload-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811374 5188 cpu_manager.go:414] "containerMap: removing container" podUID="962b779d-da15-4513-ade5-b5a289b411dd" containerName="payload-pre-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811381 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962b779d-da15-4513-ade5-b5a289b411dd" containerName="payload-pre-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811403 5188 cpu_manager.go:414] "containerMap: removing container" podUID="82aa5bfd-7b7d-435f-a76f-36c14c0f0251" containerName="payload-pre-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811411 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82aa5bfd-7b7d-435f-a76f-36c14c0f0251" containerName="payload-pre-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811418 5188 cpu_manager.go:414] "containerMap: removing container" podUID="38841d38-152d-4315-a8b8-7eba09d98ebf" containerName="maas-api" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811423 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38841d38-152d-4315-a8b8-7eba09d98ebf" containerName="maas-api" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811437 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7f20f387-c81d-41eb-804b-6b992484f56b" containerName="payload-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811441 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f20f387-c81d-41eb-804b-6b992484f56b" containerName="payload-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811606 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f20f387-c81d-41eb-804b-6b992484f56b" containerName="payload-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811628 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a00a2fc-729d-4fcb-961b-812f90b42430" containerName="maas-api" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811635 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="82289a34-8b1f-42cf-b716-f21732bcf54f" containerName="payload-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811643 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="962b779d-da15-4513-ade5-b5a289b411dd" containerName="payload-pre-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811651 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="82aa5bfd-7b7d-435f-a76f-36c14c0f0251" containerName="payload-pre-processing" Oct 31 21:44:04.811649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.811657 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="38841d38-152d-4315-a8b8-7eba09d98ebf" containerName="maas-api" Oct 31 21:44:04.816063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.816039 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:04.840647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.840614 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-09f39e-serving-cert\"" Oct 31 21:44:04.862389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.849071 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn"] Oct 31 21:44:04.898139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.898098 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rk6z\" (UniqueName: \"kubernetes.io/projected/7145f155-06c6-4262-a54e-ed2143c7640d-kube-api-access-6rk6z\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:04.898805 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.898295 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-metrics-certs\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:04.898805 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.898400 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7145f155-06c6-4262-a54e-ed2143c7640d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:04.898805 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:04.898466 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.000276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.000229 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rk6z\" (UniqueName: \"kubernetes.io/projected/7145f155-06c6-4262-a54e-ed2143c7640d-kube-api-access-6rk6z\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.000523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.000386 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-metrics-certs\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.000523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.000443 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7145f155-06c6-4262-a54e-ed2143c7640d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.000523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.000499 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.001385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.001358 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7145f155-06c6-4262-a54e-ed2143c7640d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.003660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.003638 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.003726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.003697 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-metrics-certs\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.022016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.021981 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rk6z\" (UniqueName: \"kubernetes.io/projected/7145f155-06c6-4262-a54e-ed2143c7640d-kube-api-access-6rk6z\") pod \"maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.134470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.134437 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:05.443112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:05.443083 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn"] Oct 31 21:44:05.443908 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:44:05.443879 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7145f155_06c6_4262_a54e_ed2143c7640d.slice/crio-5c9b404b027f88b204bfc7007c0710dfee154b34951ee2a70e7bafdd33000d6d WatchSource:0}: Error finding container 5c9b404b027f88b204bfc7007c0710dfee154b34951ee2a70e7bafdd33000d6d: Status 404 returned error can't find the container with id 5c9b404b027f88b204bfc7007c0710dfee154b34951ee2a70e7bafdd33000d6d Oct 31 21:44:06.178475 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:06.178435 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" event={"ID":"7145f155-06c6-4262-a54e-ed2143c7640d","Type":"ContainerStarted","Data":"07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67"} Oct 31 21:44:06.178923 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:06.178492 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" event={"ID":"7145f155-06c6-4262-a54e-ed2143c7640d","Type":"ContainerStarted","Data":"5c9b404b027f88b204bfc7007c0710dfee154b34951ee2a70e7bafdd33000d6d"} Oct 31 21:44:06.178923 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:06.178512 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:06.219744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:06.219689 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" podStartSLOduration=1.9012059030000001 podStartE2EDuration="2.21967389s" totalImagesPullingTime="318.467987ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:44:06.215517378 +0000 UTC m=+1692.908474772" watchObservedRunningTime="2026-10-31 21:44:06.21967389 +0000 UTC m=+1692.912631282" Oct 31 21:44:06.415220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:06.415177 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4"] Oct 31 21:44:06.415505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:06.415475 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" containerID="cri-o://64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871" gracePeriod=30 Oct 31 21:44:06.418941 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:06.418915 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592"] Oct 31 21:44:06.419121 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:06.419095 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" containerID="cri-o://bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8" gracePeriod=30 Oct 31 21:44:11.734348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.734300 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:44:11.764587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.764561 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:44:11.842079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842041 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-envoy\") pod \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " Oct 31 21:44:11.842271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842145 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-credential-socket\") pod \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " Oct 31 21:44:11.842271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842233 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-socket\") pod \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " Oct 31 21:44:11.842465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842282 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-kube-api-access-kpxv5\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-kube-api-access-kpxv5\") pod \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " Oct 31 21:44:11.842465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842378 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-data\") pod \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " Oct 31 21:44:11.842465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842424 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-certs\") pod \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " Oct 31 21:44:11.842629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842524 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-podinfo\") pod \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " Oct 31 21:44:11.842629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842532 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-socket" pod "67b0e627-0cf3-49cc-b08a-136f07cc07fb" (UID: "67b0e627-0cf3-49cc-b08a-136f07cc07fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.842629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842568 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istiod-ca-cert\") pod \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " Oct 31 21:44:11.842629 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842596 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-token\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-token\") pod \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\" (UID: \"67b0e627-0cf3-49cc-b08a-136f07cc07fb\") " Oct 31 21:44:11.842852 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842744 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-certs" pod "67b0e627-0cf3-49cc-b08a-136f07cc07fb" (UID: "67b0e627-0cf3-49cc-b08a-136f07cc07fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.843781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.842944 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-data" pod "67b0e627-0cf3-49cc-b08a-136f07cc07fb" (UID: "67b0e627-0cf3-49cc-b08a-136f07cc07fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.843781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.843067 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-credential-socket" pod "67b0e627-0cf3-49cc-b08a-136f07cc07fb" (UID: "67b0e627-0cf3-49cc-b08a-136f07cc07fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.843781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.843259 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istiod-ca-cert" pod "67b0e627-0cf3-49cc-b08a-136f07cc07fb" (UID: "67b0e627-0cf3-49cc-b08a-136f07cc07fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:11.843781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.843436 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.843781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.843463 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.843781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.843477 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.843781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.843489 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.843781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.843501 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.845651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.845623 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-token" pod "67b0e627-0cf3-49cc-b08a-136f07cc07fb" (UID: "67b0e627-0cf3-49cc-b08a-136f07cc07fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:11.846365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.846308 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-podinfo" pod "67b0e627-0cf3-49cc-b08a-136f07cc07fb" (UID: "67b0e627-0cf3-49cc-b08a-136f07cc07fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:44:11.846507 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.846472 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-kube-api-access-kpxv5" pod "67b0e627-0cf3-49cc-b08a-136f07cc07fb" (UID: "67b0e627-0cf3-49cc-b08a-136f07cc07fb"). InnerVolumeSpecName "kube-api-access-kpxv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:11.846685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.846666 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-envoy" pod "67b0e627-0cf3-49cc-b08a-136f07cc07fb" (UID: "67b0e627-0cf3-49cc-b08a-136f07cc07fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944021 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-istio-token\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-istio-token\") pod \"7a833011-9598-4e65-ac99-590661134d34\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944110 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-envoy\") pod \"7a833011-9598-4e65-ac99-590661134d34\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944224 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a833011-9598-4e65-ac99-590661134d34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a833011-9598-4e65-ac99-590661134d34-istiod-ca-cert\") pod \"7a833011-9598-4e65-ac99-590661134d34\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944278 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-certs\") pod \"7a833011-9598-4e65-ac99-590661134d34\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944416 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a833011-9598-4e65-ac99-590661134d34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a833011-9598-4e65-ac99-590661134d34-istio-podinfo\") pod \"7a833011-9598-4e65-ac99-590661134d34\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944451 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-data\") pod \"7a833011-9598-4e65-ac99-590661134d34\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944477 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-kube-api-access-kfpt9\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-kube-api-access-kfpt9\") pod \"7a833011-9598-4e65-ac99-590661134d34\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944546 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-socket\") pod \"7a833011-9598-4e65-ac99-590661134d34\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944577 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-credential-socket\") pod \"7a833011-9598-4e65-ac99-590661134d34\" (UID: \"7a833011-9598-4e65-ac99-590661134d34\") " Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.944779 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a833011-9598-4e65-ac99-590661134d34-istiod-ca-cert" pod "7a833011-9598-4e65-ac99-590661134d34" (UID: "7a833011-9598-4e65-ac99-590661134d34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.945107 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-data" pod "7a833011-9598-4e65-ac99-590661134d34" (UID: "7a833011-9598-4e65-ac99-590661134d34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.945182 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-credential-socket" pod "7a833011-9598-4e65-ac99-590661134d34" (UID: "7a833011-9598-4e65-ac99-590661134d34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.945245 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.945262 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a833011-9598-4e65-ac99-590661134d34-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.945272 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpxv5\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-kube-api-access-kpxv5\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.945284 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.945296 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.945310 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b0e627-0cf3-49cc-b08a-136f07cc07fb-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.945911 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-certs" pod "7a833011-9598-4e65-ac99-590661134d34" (UID: "7a833011-9598-4e65-ac99-590661134d34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.946648 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-socket" pod "7a833011-9598-4e65-ac99-590661134d34" (UID: "7a833011-9598-4e65-ac99-590661134d34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.947472 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-istio-token" pod "7a833011-9598-4e65-ac99-590661134d34" (UID: "7a833011-9598-4e65-ac99-590661134d34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.947822 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-envoy" pod "7a833011-9598-4e65-ac99-590661134d34" (UID: "7a833011-9598-4e65-ac99-590661134d34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.948129 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-kube-api-access-kfpt9" pod "7a833011-9598-4e65-ac99-590661134d34" (UID: "7a833011-9598-4e65-ac99-590661134d34"). InnerVolumeSpecName "kube-api-access-kfpt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:11.950350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:11.948584 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a833011-9598-4e65-ac99-590661134d34-istio-podinfo" pod "7a833011-9598-4e65-ac99-590661134d34" (UID: "7a833011-9598-4e65-ac99-590661134d34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:44:12.046027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.045942 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.046027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.045973 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a833011-9598-4e65-ac99-590661134d34-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.046027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.045985 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfpt9\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-kube-api-access-kfpt9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.046027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.045994 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.046027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.046003 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.046027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.046011 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a833011-9598-4e65-ac99-590661134d34-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.046027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.046019 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a833011-9598-4e65-ac99-590661134d34-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.197033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.197000 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:44:12.232244 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.232213 5188 generic.go:356] "Generic (PLEG): container finished" podID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerID="64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871" exitCode=0 Oct 31 21:44:12.232449 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.232258 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" Oct 31 21:44:12.232449 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.232299 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" event={"ID":"67b0e627-0cf3-49cc-b08a-136f07cc07fb","Type":"ContainerDied","Data":"64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871"} Oct 31 21:44:12.232449 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.232373 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4" event={"ID":"67b0e627-0cf3-49cc-b08a-136f07cc07fb","Type":"ContainerDied","Data":"c507647c75f33cf98a4e12b7acd8a747fda53e4c7aed48ccd4fd040665d5be66"} Oct 31 21:44:12.232449 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.232396 5188 scope.go:122] "RemoveContainer" containerID="64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871" Oct 31 21:44:12.233778 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.233752 5188 generic.go:356] "Generic (PLEG): container finished" podID="7a833011-9598-4e65-ac99-590661134d34" containerID="bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8" exitCode=0 Oct 31 21:44:12.233889 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.233800 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" Oct 31 21:44:12.233889 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.233835 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" event={"ID":"7a833011-9598-4e65-ac99-590661134d34","Type":"ContainerDied","Data":"bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8"} Oct 31 21:44:12.233889 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.233881 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592" event={"ID":"7a833011-9598-4e65-ac99-590661134d34","Type":"ContainerDied","Data":"535ec8f8291176a0743ee84fdf316c6c8bd218deafe4ddfbd340e6111f48b51e"} Oct 31 21:44:12.246700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.246680 5188 scope.go:122] "RemoveContainer" containerID="64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871" Oct 31 21:44:12.247048 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:44:12.247027 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871\": container with ID starting with 64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871 not found: ID does not exist" containerID="64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871" Oct 31 21:44:12.247140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.247054 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871"} err="failed to get container status \"64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871\": rpc error: code = NotFound desc = could not find container \"64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871\": container with ID starting with 64e6c65625eb8e7e2d55e084abf6320b204dbc831278be06141fd616f9ac5871 not found: ID does not exist" Oct 31 21:44:12.247140 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.247072 5188 scope.go:122] "RemoveContainer" containerID="bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8" Oct 31 21:44:12.260741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.260716 5188 scope.go:122] "RemoveContainer" containerID="bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8" Oct 31 21:44:12.262280 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:44:12.262248 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8\": container with ID starting with bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8 not found: ID does not exist" containerID="bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8" Oct 31 21:44:12.262447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.262292 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8"} err="failed to get container status \"bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8\": rpc error: code = NotFound desc = could not find container \"bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8\": container with ID starting with bfd4f59d39d1267bdb8280016583190fc672aaa4688bc37ca94067524eeecce8 not found: ID does not exist" Oct 31 21:44:12.300892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.300664 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4"] Oct 31 21:44:12.305800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.305768 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bjtx4"] Oct 31 21:44:12.334630 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.334587 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592"] Oct 31 21:44:12.340239 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:12.340206 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-np592"] Oct 31 21:44:13.475924 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:13.475883 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" path="/var/lib/kubelet/pods/67b0e627-0cf3-49cc-b08a-136f07cc07fb/volumes" Oct 31 21:44:13.476843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:13.476809 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a833011-9598-4e65-ac99-590661134d34" path="/var/lib/kubelet/pods/7a833011-9598-4e65-ac99-590661134d34/volumes" Oct 31 21:44:19.053200 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:19.053162 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x"] Oct 31 21:44:22.080444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:22.080403 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk"] Oct 31 21:44:22.116126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:22.116078 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp"] Oct 31 21:44:24.054234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.054182 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" podUID="265e13d7-4e0b-4989-96e9-779d4e19ba6c" containerName="payload-pre-processing" containerID="cri-o://54e1b73e047e46e988147fe16c47182c4216b033953f8b72324f157ede091773" gracePeriod=25 Oct 31 21:44:24.331773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.331675 5188 generic.go:356] "Generic (PLEG): container finished" podID="265e13d7-4e0b-4989-96e9-779d4e19ba6c" containerID="54e1b73e047e46e988147fe16c47182c4216b033953f8b72324f157ede091773" exitCode=0 Oct 31 21:44:24.331773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.331750 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" event={"ID":"265e13d7-4e0b-4989-96e9-779d4e19ba6c","Type":"ContainerDied","Data":"54e1b73e047e46e988147fe16c47182c4216b033953f8b72324f157ede091773"} Oct 31 21:44:24.374954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.374919 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:44:24.514911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.514864 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265e13d7-4e0b-4989-96e9-779d4e19ba6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265e13d7-4e0b-4989-96e9-779d4e19ba6c-plugins-config-volume\") pod \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\" (UID: \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\") " Oct 31 21:44:24.515101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.515027 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265e13d7-4e0b-4989-96e9-779d4e19ba6c-kube-api-access-sfq7t\" (UniqueName: \"kubernetes.io/projected/265e13d7-4e0b-4989-96e9-779d4e19ba6c-kube-api-access-sfq7t\") pod \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\" (UID: \"265e13d7-4e0b-4989-96e9-779d4e19ba6c\") " Oct 31 21:44:24.515286 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.515261 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265e13d7-4e0b-4989-96e9-779d4e19ba6c-plugins-config-volume" pod "265e13d7-4e0b-4989-96e9-779d4e19ba6c" (UID: "265e13d7-4e0b-4989-96e9-779d4e19ba6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:24.515659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.515630 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/265e13d7-4e0b-4989-96e9-779d4e19ba6c-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:24.517944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.517476 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265e13d7-4e0b-4989-96e9-779d4e19ba6c-kube-api-access-sfq7t" pod "265e13d7-4e0b-4989-96e9-779d4e19ba6c" (UID: "265e13d7-4e0b-4989-96e9-779d4e19ba6c"). InnerVolumeSpecName "kube-api-access-sfq7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:24.616857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:24.616806 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfq7t\" (UniqueName: \"kubernetes.io/projected/265e13d7-4e0b-4989-96e9-779d4e19ba6c-kube-api-access-sfq7t\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:25.341156 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:25.341130 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" Oct 31 21:44:25.341682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:25.341122 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x" event={"ID":"265e13d7-4e0b-4989-96e9-779d4e19ba6c","Type":"ContainerDied","Data":"417044a9260744734d87ce9dde2a8a78d361b9e933182fe407a9ede055551de7"} Oct 31 21:44:25.341682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:25.341284 5188 scope.go:122] "RemoveContainer" containerID="54e1b73e047e46e988147fe16c47182c4216b033953f8b72324f157ede091773" Oct 31 21:44:25.388009 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:25.387967 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x"] Oct 31 21:44:25.397136 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:25.397096 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-bb9990-7fdf9fc84c-8lc6x"] Oct 31 21:44:25.473577 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:25.473538 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265e13d7-4e0b-4989-96e9-779d4e19ba6c" path="/var/lib/kubelet/pods/265e13d7-4e0b-4989-96e9-779d4e19ba6c/volumes" Oct 31 21:44:27.081116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.081068 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" podUID="6dc824cf-e9da-4003-be08-c139b49d1652" containerName="payload-processing" containerID="cri-o://112f7fb59f2f688ae95c8766a7a5544487586785102200d1926432d07a983b8f" gracePeriod=25 Oct 31 21:44:27.117491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.117417 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" podUID="2e83a0d0-2433-4697-8779-6334653be0d9" containerName="payload-pre-processing" containerID="cri-o://ef44d366da37811eaee6960c5671a3528d857f97c17b36ee06a83743b4730f87" gracePeriod=25 Oct 31 21:44:27.359819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.359734 5188 generic.go:356] "Generic (PLEG): container finished" podID="2e83a0d0-2433-4697-8779-6334653be0d9" containerID="ef44d366da37811eaee6960c5671a3528d857f97c17b36ee06a83743b4730f87" exitCode=0 Oct 31 21:44:27.359973 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.359848 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" event={"ID":"2e83a0d0-2433-4697-8779-6334653be0d9","Type":"ContainerDied","Data":"ef44d366da37811eaee6960c5671a3528d857f97c17b36ee06a83743b4730f87"} Oct 31 21:44:27.361536 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.361509 5188 generic.go:356] "Generic (PLEG): container finished" podID="6dc824cf-e9da-4003-be08-c139b49d1652" containerID="112f7fb59f2f688ae95c8766a7a5544487586785102200d1926432d07a983b8f" exitCode=0 Oct 31 21:44:27.361675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.361558 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" event={"ID":"6dc824cf-e9da-4003-be08-c139b49d1652","Type":"ContainerDied","Data":"112f7fb59f2f688ae95c8766a7a5544487586785102200d1926432d07a983b8f"} Oct 31 21:44:27.421944 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.421916 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:44:27.500682 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.500647 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:44:27.554014 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.553975 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc824cf-e9da-4003-be08-c139b49d1652-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc824cf-e9da-4003-be08-c139b49d1652-plugins-config-volume\") pod \"6dc824cf-e9da-4003-be08-c139b49d1652\" (UID: \"6dc824cf-e9da-4003-be08-c139b49d1652\") " Oct 31 21:44:27.554200 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.554077 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e83a0d0-2433-4697-8779-6334653be0d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e83a0d0-2433-4697-8779-6334653be0d9-plugins-config-volume\") pod \"2e83a0d0-2433-4697-8779-6334653be0d9\" (UID: \"2e83a0d0-2433-4697-8779-6334653be0d9\") " Oct 31 21:44:27.554297 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.554210 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc824cf-e9da-4003-be08-c139b49d1652-kube-api-access-lh8nj\" (UniqueName: \"kubernetes.io/projected/6dc824cf-e9da-4003-be08-c139b49d1652-kube-api-access-lh8nj\") pod \"6dc824cf-e9da-4003-be08-c139b49d1652\" (UID: \"6dc824cf-e9da-4003-be08-c139b49d1652\") " Oct 31 21:44:27.554297 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.554278 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e83a0d0-2433-4697-8779-6334653be0d9-kube-api-access-rkm4g\" (UniqueName: \"kubernetes.io/projected/2e83a0d0-2433-4697-8779-6334653be0d9-kube-api-access-rkm4g\") pod \"2e83a0d0-2433-4697-8779-6334653be0d9\" (UID: \"2e83a0d0-2433-4697-8779-6334653be0d9\") " Oct 31 21:44:27.554529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.554500 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc824cf-e9da-4003-be08-c139b49d1652-plugins-config-volume" pod "6dc824cf-e9da-4003-be08-c139b49d1652" (UID: "6dc824cf-e9da-4003-be08-c139b49d1652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:27.554604 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.554513 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e83a0d0-2433-4697-8779-6334653be0d9-plugins-config-volume" pod "2e83a0d0-2433-4697-8779-6334653be0d9" (UID: "2e83a0d0-2433-4697-8779-6334653be0d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:27.554908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.554879 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e83a0d0-2433-4697-8779-6334653be0d9-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:27.554991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.554910 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc824cf-e9da-4003-be08-c139b49d1652-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:27.557139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.557116 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc824cf-e9da-4003-be08-c139b49d1652-kube-api-access-lh8nj" pod "6dc824cf-e9da-4003-be08-c139b49d1652" (UID: "6dc824cf-e9da-4003-be08-c139b49d1652"). InnerVolumeSpecName "kube-api-access-lh8nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:27.557248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.557148 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e83a0d0-2433-4697-8779-6334653be0d9-kube-api-access-rkm4g" pod "2e83a0d0-2433-4697-8779-6334653be0d9" (UID: "2e83a0d0-2433-4697-8779-6334653be0d9"). InnerVolumeSpecName "kube-api-access-rkm4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:27.655862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.655826 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh8nj\" (UniqueName: \"kubernetes.io/projected/6dc824cf-e9da-4003-be08-c139b49d1652-kube-api-access-lh8nj\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:27.655862 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:27.655856 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkm4g\" (UniqueName: \"kubernetes.io/projected/2e83a0d0-2433-4697-8779-6334653be0d9-kube-api-access-rkm4g\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:28.380162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.380131 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" Oct 31 21:44:28.380162 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.380132 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk" event={"ID":"6dc824cf-e9da-4003-be08-c139b49d1652","Type":"ContainerDied","Data":"3b3682f1c1b0c8c07c2cad16c73057f6cfd39dc84228c00c902ac311cb4189b6"} Oct 31 21:44:28.380813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.380242 5188 scope.go:122] "RemoveContainer" containerID="112f7fb59f2f688ae95c8766a7a5544487586785102200d1926432d07a983b8f" Oct 31 21:44:28.387012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.386968 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" event={"ID":"2e83a0d0-2433-4697-8779-6334653be0d9","Type":"ContainerDied","Data":"55a3ca74730766db21f69f0a5f7bcc6b48e9afb00728985cd15649e1b3048ae1"} Oct 31 21:44:28.387191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.387017 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp" Oct 31 21:44:28.394230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.394177 5188 scope.go:122] "RemoveContainer" containerID="ef44d366da37811eaee6960c5671a3528d857f97c17b36ee06a83743b4730f87" Oct 31 21:44:28.440416 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.440368 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk"] Oct 31 21:44:28.449731 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.449693 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-1d8f7b-5b7c7fc8bc-654pk"] Oct 31 21:44:28.479512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.479473 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp"] Oct 31 21:44:28.482937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.482907 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1d8f7b-8d6b4c9b7-mnbgp"] Oct 31 21:44:28.805808 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.805731 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll"] Oct 31 21:44:28.806029 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:28.805996 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" podUID="dd46ab77-7edd-408a-85b2-4bbce7bb4df0" containerName="maas-api" containerID="cri-o://5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a" gracePeriod=30 Oct 31 21:44:29.119872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.119837 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:44:29.291394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.277283 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-kube-api-access-2cbx8\") pod \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " Oct 31 21:44:29.291394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.277391 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-metrics-certs\") pod \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " Oct 31 21:44:29.291394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.277495 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-maas-api-tls\") pod \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " Oct 31 21:44:29.291394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.277592 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-openshift-service-ca\") pod \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\" (UID: \"dd46ab77-7edd-408a-85b2-4bbce7bb4df0\") " Oct 31 21:44:29.291394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.279879 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-openshift-service-ca" pod "dd46ab77-7edd-408a-85b2-4bbce7bb4df0" (UID: "dd46ab77-7edd-408a-85b2-4bbce7bb4df0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:29.291740 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.291541 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-maas-api-tls" pod "dd46ab77-7edd-408a-85b2-4bbce7bb4df0" (UID: "dd46ab77-7edd-408a-85b2-4bbce7bb4df0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:29.292026 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.291890 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-metrics-certs" pod "dd46ab77-7edd-408a-85b2-4bbce7bb4df0" (UID: "dd46ab77-7edd-408a-85b2-4bbce7bb4df0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:29.296763 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.296519 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-kube-api-access-2cbx8" pod "dd46ab77-7edd-408a-85b2-4bbce7bb4df0" (UID: "dd46ab77-7edd-408a-85b2-4bbce7bb4df0"). InnerVolumeSpecName "kube-api-access-2cbx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:29.378836 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.378796 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:29.378836 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.378824 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:29.378836 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.378834 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:29.378836 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.378845 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbx8\" (UniqueName: \"kubernetes.io/projected/dd46ab77-7edd-408a-85b2-4bbce7bb4df0-kube-api-access-2cbx8\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:29.398988 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.398942 5188 generic.go:356] "Generic (PLEG): container finished" podID="dd46ab77-7edd-408a-85b2-4bbce7bb4df0" containerID="5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a" exitCode=0 Oct 31 21:44:29.399534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.399052 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" event={"ID":"dd46ab77-7edd-408a-85b2-4bbce7bb4df0","Type":"ContainerDied","Data":"5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a"} Oct 31 21:44:29.399534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.399099 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" event={"ID":"dd46ab77-7edd-408a-85b2-4bbce7bb4df0","Type":"ContainerDied","Data":"734a3a281f868dc2ed8d9eba6411375c1f8ef95f2607a934ec5a5dc1ee0c935a"} Oct 31 21:44:29.399534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.399121 5188 scope.go:122] "RemoveContainer" containerID="5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a" Oct 31 21:44:29.399534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.399197 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll" Oct 31 21:44:29.416690 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.416661 5188 scope.go:122] "RemoveContainer" containerID="5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a" Oct 31 21:44:29.417057 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:44:29.417026 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a\": container with ID starting with 5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a not found: ID does not exist" containerID="5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a" Oct 31 21:44:29.417170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.417069 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a"} err="failed to get container status \"5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a\": rpc error: code = NotFound desc = could not find container \"5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a\": container with ID starting with 5d32e26317dbaca2390d6ce0c6dcd1b832f9289ce13f831a05237620a17a1b1a not found: ID does not exist" Oct 31 21:44:29.450293 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.450261 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll"] Oct 31 21:44:29.463168 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.462439 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bb9990-65c8c87f89-58dll"] Oct 31 21:44:29.473819 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.473776 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e83a0d0-2433-4697-8779-6334653be0d9" path="/var/lib/kubelet/pods/2e83a0d0-2433-4697-8779-6334653be0d9/volumes" Oct 31 21:44:29.474346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.474310 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc824cf-e9da-4003-be08-c139b49d1652" path="/var/lib/kubelet/pods/6dc824cf-e9da-4003-be08-c139b49d1652/volumes" Oct 31 21:44:29.474874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:29.474851 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd46ab77-7edd-408a-85b2-4bbce7bb4df0" path="/var/lib/kubelet/pods/dd46ab77-7edd-408a-85b2-4bbce7bb4df0/volumes" Oct 31 21:44:32.542632 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:44:32.542589 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:44:36.670785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:36.670751 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh"] Oct 31 21:44:36.671183 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:36.670969 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" containerID="cri-o://12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad" gracePeriod=30 Oct 31 21:44:41.977994 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:41.977960 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:44:42.140218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140174 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-certs\") pod \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " Oct 31 21:44:42.140218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140219 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-podinfo\") pod \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " Oct 31 21:44:42.140473 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140291 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istiod-ca-cert\") pod \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " Oct 31 21:44:42.140473 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140422 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-envoy\") pod \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " Oct 31 21:44:42.140579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140504 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-socket\") pod \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " Oct 31 21:44:42.140579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140555 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-credential-socket\") pod \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " Oct 31 21:44:42.140579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140566 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-certs" pod "58ba141e-72e4-4a29-be5c-fa7c9dae6317" (UID: "58ba141e-72e4-4a29-be5c-fa7c9dae6317"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:42.140744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140630 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-token\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-token\") pod \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " Oct 31 21:44:42.140744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140716 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-data\") pod \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " Oct 31 21:44:42.140866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140748 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-kube-api-access-9n5cc\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-kube-api-access-9n5cc\") pod \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\" (UID: \"58ba141e-72e4-4a29-be5c-fa7c9dae6317\") " Oct 31 21:44:42.140866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140838 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-credential-socket" pod "58ba141e-72e4-4a29-be5c-fa7c9dae6317" (UID: "58ba141e-72e4-4a29-be5c-fa7c9dae6317"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:42.140979 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140858 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istiod-ca-cert" pod "58ba141e-72e4-4a29-be5c-fa7c9dae6317" (UID: "58ba141e-72e4-4a29-be5c-fa7c9dae6317"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:42.140979 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.140859 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-socket" pod "58ba141e-72e4-4a29-be5c-fa7c9dae6317" (UID: "58ba141e-72e4-4a29-be5c-fa7c9dae6317"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:42.141090 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.141016 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-data" pod "58ba141e-72e4-4a29-be5c-fa7c9dae6317" (UID: "58ba141e-72e4-4a29-be5c-fa7c9dae6317"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:42.141304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.141282 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.141430 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.141309 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.141430 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.141346 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.141430 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.141361 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.141430 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.141373 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.142876 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.142829 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-envoy" pod "58ba141e-72e4-4a29-be5c-fa7c9dae6317" (UID: "58ba141e-72e4-4a29-be5c-fa7c9dae6317"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:42.142963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.142907 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-token" pod "58ba141e-72e4-4a29-be5c-fa7c9dae6317" (UID: "58ba141e-72e4-4a29-be5c-fa7c9dae6317"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:42.143004 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.142982 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-podinfo" pod "58ba141e-72e4-4a29-be5c-fa7c9dae6317" (UID: "58ba141e-72e4-4a29-be5c-fa7c9dae6317"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:44:42.143056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.143001 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-kube-api-access-9n5cc" pod "58ba141e-72e4-4a29-be5c-fa7c9dae6317" (UID: "58ba141e-72e4-4a29-be5c-fa7c9dae6317"). InnerVolumeSpecName "kube-api-access-9n5cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:42.242384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.242275 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.242384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.242317 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n5cc\" (UniqueName: \"kubernetes.io/projected/58ba141e-72e4-4a29-be5c-fa7c9dae6317-kube-api-access-9n5cc\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.242384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.242342 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.242384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.242351 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ba141e-72e4-4a29-be5c-fa7c9dae6317-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.516915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.516808 5188 generic.go:356] "Generic (PLEG): container finished" podID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerID="12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad" exitCode=0 Oct 31 21:44:42.516915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.516907 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" Oct 31 21:44:42.517107 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.516895 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" event={"ID":"58ba141e-72e4-4a29-be5c-fa7c9dae6317","Type":"ContainerDied","Data":"12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad"} Oct 31 21:44:42.517107 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.517028 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh" event={"ID":"58ba141e-72e4-4a29-be5c-fa7c9dae6317","Type":"ContainerDied","Data":"688a4e10c49b631ab86da329f2735c150475f3e42e9775e5a833b432f9716dc6"} Oct 31 21:44:42.517107 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.517056 5188 scope.go:122] "RemoveContainer" containerID="12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad" Oct 31 21:44:42.529400 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.529383 5188 scope.go:122] "RemoveContainer" containerID="fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57" Oct 31 21:44:42.540691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.540668 5188 scope.go:122] "RemoveContainer" containerID="12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad" Oct 31 21:44:42.541032 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:44:42.541008 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad\": container with ID starting with 12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad not found: ID does not exist" containerID="12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad" Oct 31 21:44:42.541113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.541044 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad"} err="failed to get container status \"12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad\": rpc error: code = NotFound desc = could not find container \"12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad\": container with ID starting with 12111c2976110f48cb5dbe7951b30357c7147305a4f35a7029c6578fb494ffad not found: ID does not exist" Oct 31 21:44:42.541113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.541067 5188 scope.go:122] "RemoveContainer" containerID="fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57" Oct 31 21:44:42.541376 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:44:42.541354 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57\": container with ID starting with fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57 not found: ID does not exist" containerID="fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57" Oct 31 21:44:42.541434 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.541383 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57"} err="failed to get container status \"fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57\": rpc error: code = NotFound desc = could not find container \"fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57\": container with ID starting with fadfed8102beffce0ea33d3467ad7b5935af81b18b2619305b9dab5c69ff7c57 not found: ID does not exist" Oct 31 21:44:42.571500 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.571457 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh"] Oct 31 21:44:42.575397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:42.575368 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bb9990-openshift-default-f5cbb8d6f-jqtjh"] Oct 31 21:44:43.475421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:43.475383 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" path="/var/lib/kubelet/pods/58ba141e-72e4-4a29-be5c-fa7c9dae6317/volumes" Oct 31 21:44:47.361689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.359460 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8"] Oct 31 21:44:47.361689 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.359768 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" podUID="16bdeb02-6f38-4b5f-9bc7-feaba894e996" containerName="maas-api" containerID="cri-o://95acb131353244a30ae96947f609ec714dcc048790c70318fcca318e829499ee" gracePeriod=30 Oct 31 21:44:47.570788 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.570740 5188 generic.go:356] "Generic (PLEG): container finished" podID="16bdeb02-6f38-4b5f-9bc7-feaba894e996" containerID="95acb131353244a30ae96947f609ec714dcc048790c70318fcca318e829499ee" exitCode=0 Oct 31 21:44:47.570992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.570833 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" event={"ID":"16bdeb02-6f38-4b5f-9bc7-feaba894e996","Type":"ContainerDied","Data":"95acb131353244a30ae96947f609ec714dcc048790c70318fcca318e829499ee"} Oct 31 21:44:47.662827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.662790 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:44:47.709034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.708987 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-metrics-certs\") pod \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " Oct 31 21:44:47.709034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.709040 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16bdeb02-6f38-4b5f-9bc7-feaba894e996-kube-api-access-xjbwr\" (UniqueName: \"kubernetes.io/projected/16bdeb02-6f38-4b5f-9bc7-feaba894e996-kube-api-access-xjbwr\") pod \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " Oct 31 21:44:47.709304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.709076 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-maas-api-tls\") pod \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " Oct 31 21:44:47.709304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.709109 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16bdeb02-6f38-4b5f-9bc7-feaba894e996-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16bdeb02-6f38-4b5f-9bc7-feaba894e996-openshift-service-ca\") pod \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\" (UID: \"16bdeb02-6f38-4b5f-9bc7-feaba894e996\") " Oct 31 21:44:47.709671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.709642 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bdeb02-6f38-4b5f-9bc7-feaba894e996-openshift-service-ca" pod "16bdeb02-6f38-4b5f-9bc7-feaba894e996" (UID: "16bdeb02-6f38-4b5f-9bc7-feaba894e996"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:47.712317 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.712278 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-maas-api-tls" pod "16bdeb02-6f38-4b5f-9bc7-feaba894e996" (UID: "16bdeb02-6f38-4b5f-9bc7-feaba894e996"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:47.712701 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.712674 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bdeb02-6f38-4b5f-9bc7-feaba894e996-kube-api-access-xjbwr" pod "16bdeb02-6f38-4b5f-9bc7-feaba894e996" (UID: "16bdeb02-6f38-4b5f-9bc7-feaba894e996"). InnerVolumeSpecName "kube-api-access-xjbwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:47.712798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.712748 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-metrics-certs" pod "16bdeb02-6f38-4b5f-9bc7-feaba894e996" (UID: "16bdeb02-6f38-4b5f-9bc7-feaba894e996"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:47.810621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.810584 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:47.810621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.810618 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjbwr\" (UniqueName: \"kubernetes.io/projected/16bdeb02-6f38-4b5f-9bc7-feaba894e996-kube-api-access-xjbwr\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:47.810621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.810629 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16bdeb02-6f38-4b5f-9bc7-feaba894e996-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:47.810886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:47.810638 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16bdeb02-6f38-4b5f-9bc7-feaba894e996-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:44:48.579728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:48.579698 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" Oct 31 21:44:48.580190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:48.579734 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8" event={"ID":"16bdeb02-6f38-4b5f-9bc7-feaba894e996","Type":"ContainerDied","Data":"3a2acf2c5b19d573fde070d56a1017f726c37136a6920a225275f09613621088"} Oct 31 21:44:48.580190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:48.579804 5188 scope.go:122] "RemoveContainer" containerID="95acb131353244a30ae96947f609ec714dcc048790c70318fcca318e829499ee" Oct 31 21:44:48.626274 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:48.626243 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8"] Oct 31 21:44:48.637809 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:48.637777 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1d8f7b-648d7cbdc-zs5l8"] Oct 31 21:44:49.474224 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:49.474187 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16bdeb02-6f38-4b5f-9bc7-feaba894e996" path="/var/lib/kubelet/pods/16bdeb02-6f38-4b5f-9bc7-feaba894e996/volumes" Oct 31 21:44:58.524351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.524293 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp"] Oct 31 21:44:58.525245 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525220 5188 cpu_manager.go:414] "containerMap: removing container" podUID="265e13d7-4e0b-4989-96e9-779d4e19ba6c" containerName="payload-pre-processing" Oct 31 21:44:58.525351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525246 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265e13d7-4e0b-4989-96e9-779d4e19ba6c" containerName="payload-pre-processing" Oct 31 21:44:58.525351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525273 5188 cpu_manager.go:414] "containerMap: removing container" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" Oct 31 21:44:58.525351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525281 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" Oct 31 21:44:58.525351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525291 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6dc824cf-e9da-4003-be08-c139b49d1652" containerName="payload-processing" Oct 31 21:44:58.525351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525298 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc824cf-e9da-4003-be08-c139b49d1652" containerName="payload-processing" Oct 31 21:44:58.525351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525313 5188 cpu_manager.go:414] "containerMap: removing container" podUID="2e83a0d0-2433-4697-8779-6334653be0d9" containerName="payload-pre-processing" Oct 31 21:44:58.525351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525336 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e83a0d0-2433-4697-8779-6334653be0d9" containerName="payload-pre-processing" Oct 31 21:44:58.525351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525351 5188 cpu_manager.go:414] "containerMap: removing container" podUID="dd46ab77-7edd-408a-85b2-4bbce7bb4df0" containerName="maas-api" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525360 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd46ab77-7edd-408a-85b2-4bbce7bb4df0" containerName="maas-api" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525383 5188 cpu_manager.go:414] "containerMap: removing container" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525390 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525399 5188 cpu_manager.go:414] "containerMap: removing container" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525406 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525425 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525432 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525461 5188 cpu_manager.go:414] "containerMap: removing container" podUID="16bdeb02-6f38-4b5f-9bc7-feaba894e996" containerName="maas-api" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525468 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16bdeb02-6f38-4b5f-9bc7-feaba894e996" containerName="maas-api" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525628 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="16bdeb02-6f38-4b5f-9bc7-feaba894e996" containerName="maas-api" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525649 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="265e13d7-4e0b-4989-96e9-779d4e19ba6c" containerName="payload-pre-processing" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525658 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b0e627-0cf3-49cc-b08a-136f07cc07fb" containerName="istio-proxy" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525669 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc824cf-e9da-4003-be08-c139b49d1652" containerName="payload-processing" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525678 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e83a0d0-2433-4697-8779-6334653be0d9" containerName="payload-pre-processing" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525687 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd46ab77-7edd-408a-85b2-4bbce7bb4df0" containerName="maas-api" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525698 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525708 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ba141e-72e4-4a29-be5c-fa7c9dae6317" containerName="istio-proxy" Oct 31 21:44:58.525756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.525717 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a833011-9598-4e65-ac99-590661134d34" containerName="istio-proxy" Oct 31 21:44:58.531971 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.531942 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:44:58.548503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.548467 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp"] Oct 31 21:44:58.553259 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.553224 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09f39e\"" Oct 31 21:44:58.553401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.553353 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09f39e-dockercfg-d2m8g\"" Oct 31 21:44:58.646571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.646537 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp\" (UID: \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:44:58.646755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.646594 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfxtl\" (UniqueName: \"kubernetes.io/projected/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-kube-api-access-rfxtl\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp\" (UID: \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:44:58.674668 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.674629 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c"] Oct 31 21:44:58.679095 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.679065 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:44:58.698653 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.698615 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c"] Oct 31 21:44:58.748232 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.748190 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp\" (UID: \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:44:58.748484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.748250 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj56k\" (UniqueName: \"kubernetes.io/projected/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-kube-api-access-hj56k\") pod \"payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c\" (UID: \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:44:58.748484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.748382 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfxtl\" (UniqueName: \"kubernetes.io/projected/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-kube-api-access-rfxtl\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp\" (UID: \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:44:58.748626 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.748540 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c\" (UID: \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:44:58.749080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.749058 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp\" (UID: \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:44:58.770191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.770159 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfxtl\" (UniqueName: \"kubernetes.io/projected/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-kube-api-access-rfxtl\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp\" (UID: \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:44:58.849302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.849197 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:44:58.849556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.849528 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c\" (UID: \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:44:58.849647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.849634 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj56k\" (UniqueName: \"kubernetes.io/projected/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-kube-api-access-hj56k\") pod \"payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c\" (UID: \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:44:58.850216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.850188 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c\" (UID: \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:44:58.872179 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.872109 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj56k\" (UniqueName: \"kubernetes.io/projected/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-kube-api-access-hj56k\") pod \"payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c\" (UID: \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:44:58.996687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:58.996647 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:44:59.192197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.192164 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp"] Oct 31 21:44:59.193152 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:44:59.193124 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa020c9c_ebcb_43e9_ab6a_ac14229b7717.slice/crio-2aea8d597db58b010d27b7707047dee7081119b701371b56b9593fe13777133f WatchSource:0}: Error finding container 2aea8d597db58b010d27b7707047dee7081119b701371b56b9593fe13777133f: Status 404 returned error can't find the container with id 2aea8d597db58b010d27b7707047dee7081119b701371b56b9593fe13777133f Oct 31 21:44:59.320978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.320949 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c"] Oct 31 21:44:59.666652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.666605 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" event={"ID":"fa020c9c-ebcb-43e9-ab6a-ac14229b7717","Type":"ContainerStarted","Data":"4ec8871675df4c95e76acd73f139f9509d476471cd385b6fc04e91b4c239d592"} Oct 31 21:44:59.667166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.666677 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" event={"ID":"fa020c9c-ebcb-43e9-ab6a-ac14229b7717","Type":"ContainerStarted","Data":"2aea8d597db58b010d27b7707047dee7081119b701371b56b9593fe13777133f"} Oct 31 21:44:59.667166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.666972 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:44:59.668910 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.668869 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" event={"ID":"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19","Type":"ContainerStarted","Data":"ca15467d8e793b4e8f85cb069e6b0ccb9c69b1c70cd11a84e24e59155ff84622"} Oct 31 21:44:59.669057 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.668938 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:44:59.669057 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.668961 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" event={"ID":"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19","Type":"ContainerStarted","Data":"63cec64027018ce13f6ddd04546d6032bb6fe7e2b1ef2943282126a8f6d7f925"} Oct 31 21:44:59.709718 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.709645 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" podStartSLOduration=1.709624934 podStartE2EDuration="1.709624934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:44:59.702949891 +0000 UTC m=+1746.395907283" watchObservedRunningTime="2026-10-31 21:44:59.709624934 +0000 UTC m=+1746.402582327" Oct 31 21:44:59.743124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:44:59.743024 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" podStartSLOduration=1.743009415 podStartE2EDuration="1.743009415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:44:59.742730038 +0000 UTC m=+1746.435687431" watchObservedRunningTime="2026-10-31 21:44:59.743009415 +0000 UTC m=+1746.435966807" Oct 31 21:45:00.149425 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.149369 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb"] Oct 31 21:45:00.154758 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.154718 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428"] Oct 31 21:45:00.154969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.154858 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:00.159469 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.159435 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:00.166976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.166951 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp"] Oct 31 21:45:00.172759 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.172731 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:00.183418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.181872 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb"] Oct 31 21:45:00.190892 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.190839 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:45:00.199404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.199373 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9"] Oct 31 21:45:00.212397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.211074 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:00.221401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.214571 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428"] Oct 31 21:45:00.221401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.221368 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp"] Oct 31 21:45:00.227807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.227519 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4"] Oct 31 21:45:00.244818 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.244782 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:00.249018 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.248939 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9"] Oct 31 21:45:00.262575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.262545 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4"] Oct 31 21:45:00.282186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.278437 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c436acde-98b5-49b9-afa4-efd29ef06c48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb\" (UID: \"c436acde-98b5-49b9-afa4-efd29ef06c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:00.282186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.278512 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7brtl\" (UniqueName: \"kubernetes.io/projected/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-kube-api-access-7brtl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428\" (UID: \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:00.282186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.278544 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkczt\" (UniqueName: \"kubernetes.io/projected/a227657d-f456-40a6-a2ad-9595c762d4b1-kube-api-access-xkczt\") pod \"maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9\" (UID: \"a227657d-f456-40a6-a2ad-9595c762d4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:00.282186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.278574 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8tf\" (UniqueName: \"kubernetes.io/projected/5796532d-abfd-4afb-8328-7f3abd2dc06b-kube-api-access-lw8tf\") pod \"maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp\" (UID: \"5796532d-abfd-4afb-8328-7f3abd2dc06b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:00.282186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.278601 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5796532d-abfd-4afb-8328-7f3abd2dc06b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp\" (UID: \"5796532d-abfd-4afb-8328-7f3abd2dc06b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:00.282186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.278709 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6ln\" (UniqueName: \"kubernetes.io/projected/c436acde-98b5-49b9-afa4-efd29ef06c48-kube-api-access-nh6ln\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb\" (UID: \"c436acde-98b5-49b9-afa4-efd29ef06c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:00.282186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.278764 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428\" (UID: \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:00.282186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.278868 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a227657d-f456-40a6-a2ad-9595c762d4b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9\" (UID: \"a227657d-f456-40a6-a2ad-9595c762d4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:00.380219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.380173 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6ln\" (UniqueName: \"kubernetes.io/projected/c436acde-98b5-49b9-afa4-efd29ef06c48-kube-api-access-nh6ln\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb\" (UID: \"c436acde-98b5-49b9-afa4-efd29ef06c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:00.380454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.380226 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d135b26d-a1ac-474d-a5ec-147d08187c2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4\" (UID: \"d135b26d-a1ac-474d-a5ec-147d08187c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:00.380454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.380279 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428\" (UID: \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:00.381132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.380406 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a227657d-f456-40a6-a2ad-9595c762d4b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9\" (UID: \"a227657d-f456-40a6-a2ad-9595c762d4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:00.381198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.381179 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wml27\" (UniqueName: \"kubernetes.io/projected/d135b26d-a1ac-474d-a5ec-147d08187c2d-kube-api-access-wml27\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4\" (UID: \"d135b26d-a1ac-474d-a5ec-147d08187c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:00.381247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.381089 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428\" (UID: \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:00.381247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.381213 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c436acde-98b5-49b9-afa4-efd29ef06c48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb\" (UID: \"c436acde-98b5-49b9-afa4-efd29ef06c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:00.381247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.381215 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a227657d-f456-40a6-a2ad-9595c762d4b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9\" (UID: \"a227657d-f456-40a6-a2ad-9595c762d4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:00.381423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.381399 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7brtl\" (UniqueName: \"kubernetes.io/projected/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-kube-api-access-7brtl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428\" (UID: \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:00.381480 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.381449 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkczt\" (UniqueName: \"kubernetes.io/projected/a227657d-f456-40a6-a2ad-9595c762d4b1-kube-api-access-xkczt\") pod \"maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9\" (UID: \"a227657d-f456-40a6-a2ad-9595c762d4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:00.381532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.381485 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8tf\" (UniqueName: \"kubernetes.io/projected/5796532d-abfd-4afb-8328-7f3abd2dc06b-kube-api-access-lw8tf\") pod \"maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp\" (UID: \"5796532d-abfd-4afb-8328-7f3abd2dc06b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:00.381532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.381514 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5796532d-abfd-4afb-8328-7f3abd2dc06b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp\" (UID: \"5796532d-abfd-4afb-8328-7f3abd2dc06b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:00.381988 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.381966 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c436acde-98b5-49b9-afa4-efd29ef06c48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb\" (UID: \"c436acde-98b5-49b9-afa4-efd29ef06c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:00.382377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.382286 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5796532d-abfd-4afb-8328-7f3abd2dc06b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp\" (UID: \"5796532d-abfd-4afb-8328-7f3abd2dc06b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:00.402815 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.402730 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6ln\" (UniqueName: \"kubernetes.io/projected/c436acde-98b5-49b9-afa4-efd29ef06c48-kube-api-access-nh6ln\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb\" (UID: \"c436acde-98b5-49b9-afa4-efd29ef06c48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:00.404610 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.404582 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8tf\" (UniqueName: \"kubernetes.io/projected/5796532d-abfd-4afb-8328-7f3abd2dc06b-kube-api-access-lw8tf\") pod \"maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp\" (UID: \"5796532d-abfd-4afb-8328-7f3abd2dc06b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:00.404858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.404682 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkczt\" (UniqueName: \"kubernetes.io/projected/a227657d-f456-40a6-a2ad-9595c762d4b1-kube-api-access-xkczt\") pod \"maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9\" (UID: \"a227657d-f456-40a6-a2ad-9595c762d4b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:00.405820 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.405798 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7brtl\" (UniqueName: \"kubernetes.io/projected/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-kube-api-access-7brtl\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428\" (UID: \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:00.483110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.483064 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d135b26d-a1ac-474d-a5ec-147d08187c2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4\" (UID: \"d135b26d-a1ac-474d-a5ec-147d08187c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:00.483337 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.483205 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wml27\" (UniqueName: \"kubernetes.io/projected/d135b26d-a1ac-474d-a5ec-147d08187c2d-kube-api-access-wml27\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4\" (UID: \"d135b26d-a1ac-474d-a5ec-147d08187c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:00.487703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.487665 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d135b26d-a1ac-474d-a5ec-147d08187c2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4\" (UID: \"d135b26d-a1ac-474d-a5ec-147d08187c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:00.497176 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.496919 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:00.513050 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.512838 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:00.532405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.530147 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wml27\" (UniqueName: \"kubernetes.io/projected/d135b26d-a1ac-474d-a5ec-147d08187c2d-kube-api-access-wml27\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4\" (UID: \"d135b26d-a1ac-474d-a5ec-147d08187c2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:00.551688 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.551640 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:00.567580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.567531 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:00.601348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:00.586563 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:01.115302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.115258 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428"] Oct 31 21:45:01.135936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.135886 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9"] Oct 31 21:45:01.141502 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:45:01.141450 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda227657d_f456_40a6_a2ad_9595c762d4b1.slice/crio-0e782aee04122117693bbc540bc95210c22d5def5e61e8e2df3b7b82f604bb5f WatchSource:0}: Error finding container 0e782aee04122117693bbc540bc95210c22d5def5e61e8e2df3b7b82f604bb5f: Status 404 returned error can't find the container with id 0e782aee04122117693bbc540bc95210c22d5def5e61e8e2df3b7b82f604bb5f Oct 31 21:45:01.216553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.216442 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4"] Oct 31 21:45:01.234021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.233980 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb"] Oct 31 21:45:01.244106 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:45:01.242497 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc436acde_98b5_49b9_afa4_efd29ef06c48.slice/crio-0cbf043d228d63356d73e5aa14322b8da1dbb0d36190d8549b1b1113967aaffc WatchSource:0}: Error finding container 0cbf043d228d63356d73e5aa14322b8da1dbb0d36190d8549b1b1113967aaffc: Status 404 returned error can't find the container with id 0cbf043d228d63356d73e5aa14322b8da1dbb0d36190d8549b1b1113967aaffc Oct 31 21:45:01.255588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.255533 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp"] Oct 31 21:45:01.270882 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:45:01.270854 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5796532d_abfd_4afb_8328_7f3abd2dc06b.slice/crio-bf5fedc6a1903b0ed83cd0f7fae3ad8258c58ba65f7327be2a68df7b7b802289 WatchSource:0}: Error finding container bf5fedc6a1903b0ed83cd0f7fae3ad8258c58ba65f7327be2a68df7b7b802289: Status 404 returned error can't find the container with id bf5fedc6a1903b0ed83cd0f7fae3ad8258c58ba65f7327be2a68df7b7b802289 Oct 31 21:45:01.708354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.704944 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" event={"ID":"d135b26d-a1ac-474d-a5ec-147d08187c2d","Type":"ContainerStarted","Data":"dff7bcebaca2e16c45f5aefeb660e3187e67a7ffd247239735edd3f06ad48073"} Oct 31 21:45:01.708354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.707095 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" event={"ID":"c436acde-98b5-49b9-afa4-efd29ef06c48","Type":"ContainerStarted","Data":"0cbf043d228d63356d73e5aa14322b8da1dbb0d36190d8549b1b1113967aaffc"} Oct 31 21:45:01.710512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.710476 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" event={"ID":"5796532d-abfd-4afb-8328-7f3abd2dc06b","Type":"ContainerStarted","Data":"bf5fedc6a1903b0ed83cd0f7fae3ad8258c58ba65f7327be2a68df7b7b802289"} Oct 31 21:45:01.712155 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.711980 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" event={"ID":"4e3a68d7-cae6-425e-9cd9-9abf96df76b9","Type":"ContainerStarted","Data":"163d1cecd0e27e3cd33f67da434ebbc40f3b85accfd040035562d82e309fcece"} Oct 31 21:45:01.714785 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:01.714761 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" event={"ID":"a227657d-f456-40a6-a2ad-9595c762d4b1","Type":"ContainerStarted","Data":"0e782aee04122117693bbc540bc95210c22d5def5e61e8e2df3b7b82f604bb5f"} Oct 31 21:45:02.537975 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:02.537888 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:45:02.732458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.732418 5188 generic.go:356] "Generic (PLEG): container finished" podID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerID="47f7d050e3b20d7f720f8a27e34400168c23d4635128fb0df263503c284c374b" exitCode=60 Oct 31 21:45:02.732675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.732499 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" event={"ID":"c436acde-98b5-49b9-afa4-efd29ef06c48","Type":"ContainerDied","Data":"47f7d050e3b20d7f720f8a27e34400168c23d4635128fb0df263503c284c374b"} Oct 31 21:45:02.732823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.732806 5188 scope.go:122] "RemoveContainer" containerID="47f7d050e3b20d7f720f8a27e34400168c23d4635128fb0df263503c284c374b" Oct 31 21:45:02.734539 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.734503 5188 generic.go:356] "Generic (PLEG): container finished" podID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerID="22d22eb424738c5d1d825f61e6c0c41c48256506ade7f30dd9651fa6bae72203" exitCode=60 Oct 31 21:45:02.734739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.734575 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" event={"ID":"5796532d-abfd-4afb-8328-7f3abd2dc06b","Type":"ContainerDied","Data":"22d22eb424738c5d1d825f61e6c0c41c48256506ade7f30dd9651fa6bae72203"} Oct 31 21:45:02.735216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.734862 5188 scope.go:122] "RemoveContainer" containerID="22d22eb424738c5d1d825f61e6c0c41c48256506ade7f30dd9651fa6bae72203" Oct 31 21:45:02.736744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.736718 5188 generic.go:356] "Generic (PLEG): container finished" podID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerID="9eb836f8a78a35fc881de4bc87e765c9f8475a89dfa47825256602314d7897c1" exitCode=60 Oct 31 21:45:02.736912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.736847 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" event={"ID":"4e3a68d7-cae6-425e-9cd9-9abf96df76b9","Type":"ContainerDied","Data":"9eb836f8a78a35fc881de4bc87e765c9f8475a89dfa47825256602314d7897c1"} Oct 31 21:45:02.737041 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.737027 5188 scope.go:122] "RemoveContainer" containerID="9eb836f8a78a35fc881de4bc87e765c9f8475a89dfa47825256602314d7897c1" Oct 31 21:45:02.738833 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.738812 5188 generic.go:356] "Generic (PLEG): container finished" podID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerID="d1753d1aeeea6e2cda86b88aa6c01ac42622c87940a5bf1a05c92ce19c5fd0e2" exitCode=60 Oct 31 21:45:02.738935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.738895 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" event={"ID":"a227657d-f456-40a6-a2ad-9595c762d4b1","Type":"ContainerDied","Data":"d1753d1aeeea6e2cda86b88aa6c01ac42622c87940a5bf1a05c92ce19c5fd0e2"} Oct 31 21:45:02.739110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.739088 5188 scope.go:122] "RemoveContainer" containerID="d1753d1aeeea6e2cda86b88aa6c01ac42622c87940a5bf1a05c92ce19c5fd0e2" Oct 31 21:45:02.740970 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.740947 5188 generic.go:356] "Generic (PLEG): container finished" podID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerID="01a82341dd2237df1842364d28ffd2dec34779d126cb1e4f41420f5d4236494a" exitCode=60 Oct 31 21:45:02.741063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.740989 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" event={"ID":"d135b26d-a1ac-474d-a5ec-147d08187c2d","Type":"ContainerDied","Data":"01a82341dd2237df1842364d28ffd2dec34779d126cb1e4f41420f5d4236494a"} Oct 31 21:45:02.741236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:02.741221 5188 scope.go:122] "RemoveContainer" containerID="01a82341dd2237df1842364d28ffd2dec34779d126cb1e4f41420f5d4236494a" Oct 31 21:45:03.751388 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.751284 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" event={"ID":"5796532d-abfd-4afb-8328-7f3abd2dc06b","Type":"ContainerStarted","Data":"6250c6d995230f3ed7bcf43a46a44b20b8ad9bfbb52f5917583f088f0f7efc7c"} Oct 31 21:45:03.753386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.753356 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" event={"ID":"4e3a68d7-cae6-425e-9cd9-9abf96df76b9","Type":"ContainerStarted","Data":"c576f012463d70cec0aaf3c559cd9a75ec955b496cf61576f5f6b0b35551ee26"} Oct 31 21:45:03.755533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.755505 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" event={"ID":"a227657d-f456-40a6-a2ad-9595c762d4b1","Type":"ContainerStarted","Data":"c6184c52c638ec6496f0f3633d0976f0bf0e26c112cb611c3a88edba8f9ff08f"} Oct 31 21:45:03.757721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.757694 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" event={"ID":"d135b26d-a1ac-474d-a5ec-147d08187c2d","Type":"ContainerStarted","Data":"42592d5fc1fd62b000688fd0bc54292ff992017ed105f323258a579db96f8425"} Oct 31 21:45:03.759644 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.759603 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" event={"ID":"c436acde-98b5-49b9-afa4-efd29ef06c48","Type":"ContainerStarted","Data":"faed9fcd81b562494a99d4ff95e6df5356eb03056f1727210916d136914b7e60"} Oct 31 21:45:03.790663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.790611 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" podStartSLOduration=3.790596415 podStartE2EDuration="3.790596415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.787984781 +0000 UTC m=+1750.480942174" watchObservedRunningTime="2026-10-31 21:45:03.790596415 +0000 UTC m=+1750.483553806" Oct 31 21:45:03.824532 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.824470 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" podStartSLOduration=3.824455596 podStartE2EDuration="3.824455596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.823589746 +0000 UTC m=+1750.516547135" watchObservedRunningTime="2026-10-31 21:45:03.824455596 +0000 UTC m=+1750.517412989" Oct 31 21:45:03.863485 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.863435 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" podStartSLOduration=3.863422922 podStartE2EDuration="3.863422922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.862905999 +0000 UTC m=+1750.555863390" watchObservedRunningTime="2026-10-31 21:45:03.863422922 +0000 UTC m=+1750.556380313" Oct 31 21:45:03.902348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.902237 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" podStartSLOduration=3.902222563 podStartE2EDuration="3.902222563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.898293694 +0000 UTC m=+1750.591251088" watchObservedRunningTime="2026-10-31 21:45:03.902222563 +0000 UTC m=+1750.595179955" Oct 31 21:45:03.947150 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:03.947070 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" podStartSLOduration=3.947051179 podStartE2EDuration="3.947051179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.933949823 +0000 UTC m=+1750.626907214" watchObservedRunningTime="2026-10-31 21:45:03.947051179 +0000 UTC m=+1750.640008573" Oct 31 21:45:04.769427 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.769389 5188 generic.go:356] "Generic (PLEG): container finished" podID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerID="faed9fcd81b562494a99d4ff95e6df5356eb03056f1727210916d136914b7e60" exitCode=60 Oct 31 21:45:04.770005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.769480 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" event={"ID":"c436acde-98b5-49b9-afa4-efd29ef06c48","Type":"ContainerDied","Data":"faed9fcd81b562494a99d4ff95e6df5356eb03056f1727210916d136914b7e60"} Oct 31 21:45:04.770005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.769547 5188 scope.go:122] "RemoveContainer" containerID="47f7d050e3b20d7f720f8a27e34400168c23d4635128fb0df263503c284c374b" Oct 31 21:45:04.770005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.769769 5188 scope.go:122] "RemoveContainer" containerID="faed9fcd81b562494a99d4ff95e6df5356eb03056f1727210916d136914b7e60" Oct 31 21:45:04.770235 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:04.770085 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb_odh-ai-gateway-infra(c436acde-98b5-49b9-afa4-efd29ef06c48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" Oct 31 21:45:04.771599 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.771565 5188 generic.go:356] "Generic (PLEG): container finished" podID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerID="6250c6d995230f3ed7bcf43a46a44b20b8ad9bfbb52f5917583f088f0f7efc7c" exitCode=60 Oct 31 21:45:04.771745 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.771661 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" event={"ID":"5796532d-abfd-4afb-8328-7f3abd2dc06b","Type":"ContainerDied","Data":"6250c6d995230f3ed7bcf43a46a44b20b8ad9bfbb52f5917583f088f0f7efc7c"} Oct 31 21:45:04.771923 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.771908 5188 scope.go:122] "RemoveContainer" containerID="6250c6d995230f3ed7bcf43a46a44b20b8ad9bfbb52f5917583f088f0f7efc7c" Oct 31 21:45:04.772229 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:04.772204 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp_odh-ai-gateway-infra(5796532d-abfd-4afb-8328-7f3abd2dc06b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" Oct 31 21:45:04.774058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.774033 5188 generic.go:356] "Generic (PLEG): container finished" podID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerID="c576f012463d70cec0aaf3c559cd9a75ec955b496cf61576f5f6b0b35551ee26" exitCode=60 Oct 31 21:45:04.774160 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.774148 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" event={"ID":"4e3a68d7-cae6-425e-9cd9-9abf96df76b9","Type":"ContainerDied","Data":"c576f012463d70cec0aaf3c559cd9a75ec955b496cf61576f5f6b0b35551ee26"} Oct 31 21:45:04.774504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.774484 5188 scope.go:122] "RemoveContainer" containerID="c576f012463d70cec0aaf3c559cd9a75ec955b496cf61576f5f6b0b35551ee26" Oct 31 21:45:04.774749 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:04.774729 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428_odh-ai-gateway-infra(4e3a68d7-cae6-425e-9cd9-9abf96df76b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" Oct 31 21:45:04.777078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.777051 5188 generic.go:356] "Generic (PLEG): container finished" podID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerID="c6184c52c638ec6496f0f3633d0976f0bf0e26c112cb611c3a88edba8f9ff08f" exitCode=60 Oct 31 21:45:04.777184 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.777131 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" event={"ID":"a227657d-f456-40a6-a2ad-9595c762d4b1","Type":"ContainerDied","Data":"c6184c52c638ec6496f0f3633d0976f0bf0e26c112cb611c3a88edba8f9ff08f"} Oct 31 21:45:04.777780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.777758 5188 scope.go:122] "RemoveContainer" containerID="c6184c52c638ec6496f0f3633d0976f0bf0e26c112cb611c3a88edba8f9ff08f" Oct 31 21:45:04.778066 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:04.778038 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9_odh-ai-gateway-infra(a227657d-f456-40a6-a2ad-9595c762d4b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" Oct 31 21:45:04.779491 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.779470 5188 generic.go:356] "Generic (PLEG): container finished" podID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerID="42592d5fc1fd62b000688fd0bc54292ff992017ed105f323258a579db96f8425" exitCode=60 Oct 31 21:45:04.779699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.779645 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" event={"ID":"d135b26d-a1ac-474d-a5ec-147d08187c2d","Type":"ContainerDied","Data":"42592d5fc1fd62b000688fd0bc54292ff992017ed105f323258a579db96f8425"} Oct 31 21:45:04.780048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.779858 5188 scope.go:122] "RemoveContainer" containerID="42592d5fc1fd62b000688fd0bc54292ff992017ed105f323258a579db96f8425" Oct 31 21:45:04.780128 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:04.780067 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4_odh-ai-gateway-infra(d135b26d-a1ac-474d-a5ec-147d08187c2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" Oct 31 21:45:04.786499 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.786475 5188 scope.go:122] "RemoveContainer" containerID="22d22eb424738c5d1d825f61e6c0c41c48256506ade7f30dd9651fa6bae72203" Oct 31 21:45:04.810403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.810376 5188 scope.go:122] "RemoveContainer" containerID="9eb836f8a78a35fc881de4bc87e765c9f8475a89dfa47825256602314d7897c1" Oct 31 21:45:04.825764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.825740 5188 scope.go:122] "RemoveContainer" containerID="d1753d1aeeea6e2cda86b88aa6c01ac42622c87940a5bf1a05c92ce19c5fd0e2" Oct 31 21:45:04.848389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:04.848355 5188 scope.go:122] "RemoveContainer" containerID="01a82341dd2237df1842364d28ffd2dec34779d126cb1e4f41420f5d4236494a" Oct 31 21:45:05.791885 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:05.791854 5188 scope.go:122] "RemoveContainer" containerID="faed9fcd81b562494a99d4ff95e6df5356eb03056f1727210916d136914b7e60" Oct 31 21:45:05.792369 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:05.792193 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb_odh-ai-gateway-infra(c436acde-98b5-49b9-afa4-efd29ef06c48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" Oct 31 21:45:05.794036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:05.794013 5188 scope.go:122] "RemoveContainer" containerID="6250c6d995230f3ed7bcf43a46a44b20b8ad9bfbb52f5917583f088f0f7efc7c" Oct 31 21:45:05.794340 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:05.794304 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp_odh-ai-gateway-infra(5796532d-abfd-4afb-8328-7f3abd2dc06b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" Oct 31 21:45:05.795956 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:05.795933 5188 scope.go:122] "RemoveContainer" containerID="c576f012463d70cec0aaf3c559cd9a75ec955b496cf61576f5f6b0b35551ee26" Oct 31 21:45:05.796292 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:05.796266 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428_odh-ai-gateway-infra(4e3a68d7-cae6-425e-9cd9-9abf96df76b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" Oct 31 21:45:05.797900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:05.797880 5188 scope.go:122] "RemoveContainer" containerID="c6184c52c638ec6496f0f3633d0976f0bf0e26c112cb611c3a88edba8f9ff08f" Oct 31 21:45:05.798140 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:05.798113 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9_odh-ai-gateway-infra(a227657d-f456-40a6-a2ad-9595c762d4b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" Oct 31 21:45:05.799728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:05.799707 5188 scope.go:122] "RemoveContainer" containerID="42592d5fc1fd62b000688fd0bc54292ff992017ed105f323258a579db96f8425" Oct 31 21:45:05.799933 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:05.799916 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4_odh-ai-gateway-infra(d135b26d-a1ac-474d-a5ec-147d08187c2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" Oct 31 21:45:06.050154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:06.050057 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226"] Oct 31 21:45:06.089699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:06.089647 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89"] Oct 31 21:45:10.724496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:10.716211 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:45:10.724496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:10.719523 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:45:11.050787 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.050684 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" podUID="e168f19b-bf27-4c7c-90a5-a9080d8dbc4d" containerName="payload-processing" containerID="cri-o://b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f" gracePeriod=25 Oct 31 21:45:11.090734 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.090693 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" podUID="a4a78bb6-6094-454f-915a-a76064ab3b22" containerName="payload-pre-processing" containerID="cri-o://406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256" gracePeriod=25 Oct 31 21:45:11.399692 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.399660 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:45:11.447511 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.447482 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:45:11.563817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.563771 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-kube-api-access-zmdrq\" (UniqueName: \"kubernetes.io/projected/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-kube-api-access-zmdrq\") pod \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\" (UID: \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\") " Oct 31 21:45:11.564025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.563829 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a78bb6-6094-454f-915a-a76064ab3b22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a78bb6-6094-454f-915a-a76064ab3b22-plugins-config-volume\") pod \"a4a78bb6-6094-454f-915a-a76064ab3b22\" (UID: \"a4a78bb6-6094-454f-915a-a76064ab3b22\") " Oct 31 21:45:11.564025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.563870 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-plugins-config-volume\") pod \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\" (UID: \"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d\") " Oct 31 21:45:11.564025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.563948 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a78bb6-6094-454f-915a-a76064ab3b22-kube-api-access-gb6vr\" (UniqueName: \"kubernetes.io/projected/a4a78bb6-6094-454f-915a-a76064ab3b22-kube-api-access-gb6vr\") pod \"a4a78bb6-6094-454f-915a-a76064ab3b22\" (UID: \"a4a78bb6-6094-454f-915a-a76064ab3b22\") " Oct 31 21:45:11.564273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.564247 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a78bb6-6094-454f-915a-a76064ab3b22-plugins-config-volume" pod "a4a78bb6-6094-454f-915a-a76064ab3b22" (UID: "a4a78bb6-6094-454f-915a-a76064ab3b22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:11.564368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.564343 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-plugins-config-volume" pod "e168f19b-bf27-4c7c-90a5-a9080d8dbc4d" (UID: "e168f19b-bf27-4c7c-90a5-a9080d8dbc4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:11.564940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.564659 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4a78bb6-6094-454f-915a-a76064ab3b22-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:11.564940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.564756 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:11.566190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.566165 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-kube-api-access-zmdrq" pod "e168f19b-bf27-4c7c-90a5-a9080d8dbc4d" (UID: "e168f19b-bf27-4c7c-90a5-a9080d8dbc4d"). InnerVolumeSpecName "kube-api-access-zmdrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:11.566190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.566181 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a78bb6-6094-454f-915a-a76064ab3b22-kube-api-access-gb6vr" pod "a4a78bb6-6094-454f-915a-a76064ab3b22" (UID: "a4a78bb6-6094-454f-915a-a76064ab3b22"). InnerVolumeSpecName "kube-api-access-gb6vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:11.666619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.666583 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmdrq\" (UniqueName: \"kubernetes.io/projected/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d-kube-api-access-zmdrq\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:11.666619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.666616 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb6vr\" (UniqueName: \"kubernetes.io/projected/a4a78bb6-6094-454f-915a-a76064ab3b22-kube-api-access-gb6vr\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:11.851568 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.851529 5188 generic.go:356] "Generic (PLEG): container finished" podID="a4a78bb6-6094-454f-915a-a76064ab3b22" containerID="406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256" exitCode=0 Oct 31 21:45:11.852115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.851628 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" Oct 31 21:45:11.852115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.851637 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" event={"ID":"a4a78bb6-6094-454f-915a-a76064ab3b22","Type":"ContainerDied","Data":"406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256"} Oct 31 21:45:11.852115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.851717 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89" event={"ID":"a4a78bb6-6094-454f-915a-a76064ab3b22","Type":"ContainerDied","Data":"569a1e96df7b64b58b6df5fd23d2d547865fa010d6c99fe72f3477784424d93b"} Oct 31 21:45:11.852115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.851751 5188 scope.go:122] "RemoveContainer" containerID="406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256" Oct 31 21:45:11.853673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.853650 5188 generic.go:356] "Generic (PLEG): container finished" podID="e168f19b-bf27-4c7c-90a5-a9080d8dbc4d" containerID="b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f" exitCode=0 Oct 31 21:45:11.853799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.853712 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" Oct 31 21:45:11.853799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.853775 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" event={"ID":"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d","Type":"ContainerDied","Data":"b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f"} Oct 31 21:45:11.853903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.853827 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226" event={"ID":"e168f19b-bf27-4c7c-90a5-a9080d8dbc4d","Type":"ContainerDied","Data":"f1c65926fbe958222a215a97781fe546d2ce2658e8f7d5f200902dd64e9b6bc3"} Oct 31 21:45:11.863669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.863640 5188 scope.go:122] "RemoveContainer" containerID="406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256" Oct 31 21:45:11.864046 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:11.864014 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256\": container with ID starting with 406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256 not found: ID does not exist" containerID="406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256" Oct 31 21:45:11.864163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.864055 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256"} err="failed to get container status \"406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256\": rpc error: code = NotFound desc = could not find container \"406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256\": container with ID starting with 406698a28b903692ca0c9b8e6edd010e28a0820f512429ed0aa5f261f49b9256 not found: ID does not exist" Oct 31 21:45:11.864163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.864082 5188 scope.go:122] "RemoveContainer" containerID="b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f" Oct 31 21:45:11.876176 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.876153 5188 scope.go:122] "RemoveContainer" containerID="b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f" Oct 31 21:45:11.876567 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:11.876547 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f\": container with ID starting with b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f not found: ID does not exist" containerID="b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f" Oct 31 21:45:11.876628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.876577 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f"} err="failed to get container status \"b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f\": rpc error: code = NotFound desc = could not find container \"b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f\": container with ID starting with b20edba6e9f728f20bcbf1c05873168700e7451f9fc52149c1f86d2598d9345f not found: ID does not exist" Oct 31 21:45:11.913539 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.913502 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226"] Oct 31 21:45:11.923375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.923252 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f4569e-d74dfcc47-qk226"] Oct 31 21:45:11.954433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.954191 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89"] Oct 31 21:45:11.961361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:11.961308 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f4569e-6dd47f74dc-jkc89"] Oct 31 21:45:13.476780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:13.476744 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a78bb6-6094-454f-915a-a76064ab3b22" path="/var/lib/kubelet/pods/a4a78bb6-6094-454f-915a-a76064ab3b22/volumes" Oct 31 21:45:13.477288 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:13.477224 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e168f19b-bf27-4c7c-90a5-a9080d8dbc4d" path="/var/lib/kubelet/pods/e168f19b-bf27-4c7c-90a5-a9080d8dbc4d/volumes" Oct 31 21:45:16.465981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:16.465939 5188 scope.go:122] "RemoveContainer" containerID="c6184c52c638ec6496f0f3633d0976f0bf0e26c112cb611c3a88edba8f9ff08f" Oct 31 21:45:16.900562 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:16.900518 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" event={"ID":"a227657d-f456-40a6-a2ad-9595c762d4b1","Type":"ContainerStarted","Data":"c3d4472b420a9eab47cdc66ee517ac9ff3648d3e3174b32281c086cdc0174393"} Oct 31 21:45:17.466023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:17.465987 5188 scope.go:122] "RemoveContainer" containerID="42592d5fc1fd62b000688fd0bc54292ff992017ed105f323258a579db96f8425" Oct 31 21:45:17.466545 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:17.466237 5188 scope.go:122] "RemoveContainer" containerID="6250c6d995230f3ed7bcf43a46a44b20b8ad9bfbb52f5917583f088f0f7efc7c" Oct 31 21:45:17.551559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:17.551531 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c_dfdfc871-39b0-46e5-9a2b-62c08b7e2c19/payload-processing/0.log" Oct 31 21:45:17.929675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:17.929637 5188 generic.go:356] "Generic (PLEG): container finished" podID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerID="c3d4472b420a9eab47cdc66ee517ac9ff3648d3e3174b32281c086cdc0174393" exitCode=60 Oct 31 21:45:17.929874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:17.929855 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" event={"ID":"a227657d-f456-40a6-a2ad-9595c762d4b1","Type":"ContainerDied","Data":"c3d4472b420a9eab47cdc66ee517ac9ff3648d3e3174b32281c086cdc0174393"} Oct 31 21:45:17.929961 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:17.929945 5188 scope.go:122] "RemoveContainer" containerID="c6184c52c638ec6496f0f3633d0976f0bf0e26c112cb611c3a88edba8f9ff08f" Oct 31 21:45:17.930370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:17.930316 5188 scope.go:122] "RemoveContainer" containerID="c3d4472b420a9eab47cdc66ee517ac9ff3648d3e3174b32281c086cdc0174393" Oct 31 21:45:17.930842 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:17.930681 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9_odh-ai-gateway-infra(a227657d-f456-40a6-a2ad-9595c762d4b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" Oct 31 21:45:17.933284 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:17.933260 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" event={"ID":"5796532d-abfd-4afb-8328-7f3abd2dc06b","Type":"ContainerStarted","Data":"3a87c062850d730e9b73723bff351fb305e7f6f32db4be587dcd9676959dcb9a"} Oct 31 21:45:17.949735 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:17.949708 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9"] Oct 31 21:45:18.466334 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.466301 5188 scope.go:122] "RemoveContainer" containerID="faed9fcd81b562494a99d4ff95e6df5356eb03056f1727210916d136914b7e60" Oct 31 21:45:18.466803 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.466504 5188 scope.go:122] "RemoveContainer" containerID="c576f012463d70cec0aaf3c559cd9a75ec955b496cf61576f5f6b0b35551ee26" Oct 31 21:45:18.943741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.943606 5188 generic.go:356] "Generic (PLEG): container finished" podID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerID="3a87c062850d730e9b73723bff351fb305e7f6f32db4be587dcd9676959dcb9a" exitCode=60 Oct 31 21:45:18.943741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.943674 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" event={"ID":"5796532d-abfd-4afb-8328-7f3abd2dc06b","Type":"ContainerDied","Data":"3a87c062850d730e9b73723bff351fb305e7f6f32db4be587dcd9676959dcb9a"} Oct 31 21:45:18.943991 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.943749 5188 scope.go:122] "RemoveContainer" containerID="6250c6d995230f3ed7bcf43a46a44b20b8ad9bfbb52f5917583f088f0f7efc7c" Oct 31 21:45:18.944215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.944192 5188 scope.go:122] "RemoveContainer" containerID="3a87c062850d730e9b73723bff351fb305e7f6f32db4be587dcd9676959dcb9a" Oct 31 21:45:18.944628 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:18.944594 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp_odh-ai-gateway-infra(5796532d-abfd-4afb-8328-7f3abd2dc06b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" Oct 31 21:45:18.946581 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.946554 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" event={"ID":"4e3a68d7-cae6-425e-9cd9-9abf96df76b9","Type":"ContainerStarted","Data":"005c458be536400b66d348962679c01fde79dcd1872ab59017cf9fb388fef148"} Oct 31 21:45:18.951047 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.951017 5188 generic.go:356] "Generic (PLEG): container finished" podID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerID="23bb35a2f0a5c5e82443339a1a8b7ce236701e3158e273bf5dfe6684d2ca6cd6" exitCode=60 Oct 31 21:45:18.951181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.951053 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" event={"ID":"d135b26d-a1ac-474d-a5ec-147d08187c2d","Type":"ContainerDied","Data":"23bb35a2f0a5c5e82443339a1a8b7ce236701e3158e273bf5dfe6684d2ca6cd6"} Oct 31 21:45:18.951544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.951527 5188 scope.go:122] "RemoveContainer" containerID="23bb35a2f0a5c5e82443339a1a8b7ce236701e3158e273bf5dfe6684d2ca6cd6" Oct 31 21:45:18.951904 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:18.951884 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4_odh-ai-gateway-infra(d135b26d-a1ac-474d-a5ec-147d08187c2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" Oct 31 21:45:18.960216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:18.960194 5188 scope.go:122] "RemoveContainer" containerID="42592d5fc1fd62b000688fd0bc54292ff992017ed105f323258a579db96f8425" Oct 31 21:45:19.032886 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.032839 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp"] Oct 31 21:45:19.140859 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.140822 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:19.158882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.158838 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a227657d-f456-40a6-a2ad-9595c762d4b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a227657d-f456-40a6-a2ad-9595c762d4b1-maas-api-service-ca\") pod \"a227657d-f456-40a6-a2ad-9595c762d4b1\" (UID: \"a227657d-f456-40a6-a2ad-9595c762d4b1\") " Oct 31 21:45:19.158882 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.158879 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a227657d-f456-40a6-a2ad-9595c762d4b1-kube-api-access-xkczt\" (UniqueName: \"kubernetes.io/projected/a227657d-f456-40a6-a2ad-9595c762d4b1-kube-api-access-xkczt\") pod \"a227657d-f456-40a6-a2ad-9595c762d4b1\" (UID: \"a227657d-f456-40a6-a2ad-9595c762d4b1\") " Oct 31 21:45:19.159281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.159254 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a227657d-f456-40a6-a2ad-9595c762d4b1-maas-api-service-ca" pod "a227657d-f456-40a6-a2ad-9595c762d4b1" (UID: "a227657d-f456-40a6-a2ad-9595c762d4b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:19.159514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.159496 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a227657d-f456-40a6-a2ad-9595c762d4b1-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:19.161212 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.161183 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a227657d-f456-40a6-a2ad-9595c762d4b1-kube-api-access-xkczt" pod "a227657d-f456-40a6-a2ad-9595c762d4b1" (UID: "a227657d-f456-40a6-a2ad-9595c762d4b1"). InnerVolumeSpecName "kube-api-access-xkczt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:19.260844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.260735 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkczt\" (UniqueName: \"kubernetes.io/projected/a227657d-f456-40a6-a2ad-9595c762d4b1-kube-api-access-xkczt\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:19.966248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.966199 5188 generic.go:356] "Generic (PLEG): container finished" podID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerID="005c458be536400b66d348962679c01fde79dcd1872ab59017cf9fb388fef148" exitCode=60 Oct 31 21:45:19.966665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.966279 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" event={"ID":"4e3a68d7-cae6-425e-9cd9-9abf96df76b9","Type":"ContainerDied","Data":"005c458be536400b66d348962679c01fde79dcd1872ab59017cf9fb388fef148"} Oct 31 21:45:19.966665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.966404 5188 scope.go:122] "RemoveContainer" containerID="c576f012463d70cec0aaf3c559cd9a75ec955b496cf61576f5f6b0b35551ee26" Oct 31 21:45:19.966896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.966875 5188 scope.go:122] "RemoveContainer" containerID="005c458be536400b66d348962679c01fde79dcd1872ab59017cf9fb388fef148" Oct 31 21:45:19.967274 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:19.967248 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428_odh-ai-gateway-infra(4e3a68d7-cae6-425e-9cd9-9abf96df76b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" Oct 31 21:45:19.969109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.968911 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" Oct 31 21:45:19.969109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.968934 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9" event={"ID":"a227657d-f456-40a6-a2ad-9595c762d4b1","Type":"ContainerDied","Data":"0e782aee04122117693bbc540bc95210c22d5def5e61e8e2df3b7b82f604bb5f"} Oct 31 21:45:19.973857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.973835 5188 generic.go:356] "Generic (PLEG): container finished" podID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerID="7db3cc060091677f48a254af01acb1ae5347298efd8b35becfadd142509853a2" exitCode=60 Oct 31 21:45:19.973958 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.973878 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" event={"ID":"c436acde-98b5-49b9-afa4-efd29ef06c48","Type":"ContainerDied","Data":"7db3cc060091677f48a254af01acb1ae5347298efd8b35becfadd142509853a2"} Oct 31 21:45:19.974236 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.974222 5188 scope.go:122] "RemoveContainer" containerID="7db3cc060091677f48a254af01acb1ae5347298efd8b35becfadd142509853a2" Oct 31 21:45:19.974538 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:19.974519 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb_odh-ai-gateway-infra(c436acde-98b5-49b9-afa4-efd29ef06c48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" Oct 31 21:45:19.991983 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:19.991958 5188 scope.go:122] "RemoveContainer" containerID="c3d4472b420a9eab47cdc66ee517ac9ff3648d3e3174b32281c086cdc0174393" Oct 31 21:45:20.010574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.010544 5188 scope.go:122] "RemoveContainer" containerID="faed9fcd81b562494a99d4ff95e6df5356eb03056f1727210916d136914b7e60" Oct 31 21:45:20.026987 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.026949 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4"] Oct 31 21:45:20.071990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.071929 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428"] Oct 31 21:45:20.085199 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.085163 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9"] Oct 31 21:45:20.095006 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.094954 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d43b66-29891385-wmlf9"] Oct 31 21:45:20.176220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.176186 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:20.251996 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.251710 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:20.276969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.275562 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d135b26d-a1ac-474d-a5ec-147d08187c2d-kube-api-access-wml27\" (UniqueName: \"kubernetes.io/projected/d135b26d-a1ac-474d-a5ec-147d08187c2d-kube-api-access-wml27\") pod \"d135b26d-a1ac-474d-a5ec-147d08187c2d\" (UID: \"d135b26d-a1ac-474d-a5ec-147d08187c2d\") " Oct 31 21:45:20.276969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.275713 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5796532d-abfd-4afb-8328-7f3abd2dc06b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5796532d-abfd-4afb-8328-7f3abd2dc06b-maas-api-service-ca\") pod \"5796532d-abfd-4afb-8328-7f3abd2dc06b\" (UID: \"5796532d-abfd-4afb-8328-7f3abd2dc06b\") " Oct 31 21:45:20.276969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.275795 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5796532d-abfd-4afb-8328-7f3abd2dc06b-kube-api-access-lw8tf\" (UniqueName: \"kubernetes.io/projected/5796532d-abfd-4afb-8328-7f3abd2dc06b-kube-api-access-lw8tf\") pod \"5796532d-abfd-4afb-8328-7f3abd2dc06b\" (UID: \"5796532d-abfd-4afb-8328-7f3abd2dc06b\") " Oct 31 21:45:20.276969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.276020 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d135b26d-a1ac-474d-a5ec-147d08187c2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d135b26d-a1ac-474d-a5ec-147d08187c2d-maas-api-service-ca\") pod \"d135b26d-a1ac-474d-a5ec-147d08187c2d\" (UID: \"d135b26d-a1ac-474d-a5ec-147d08187c2d\") " Oct 31 21:45:20.276969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.276542 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5796532d-abfd-4afb-8328-7f3abd2dc06b-maas-api-service-ca" pod "5796532d-abfd-4afb-8328-7f3abd2dc06b" (UID: "5796532d-abfd-4afb-8328-7f3abd2dc06b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:20.276969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.276944 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d135b26d-a1ac-474d-a5ec-147d08187c2d-maas-api-service-ca" pod "d135b26d-a1ac-474d-a5ec-147d08187c2d" (UID: "d135b26d-a1ac-474d-a5ec-147d08187c2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:20.278585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.278556 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d135b26d-a1ac-474d-a5ec-147d08187c2d-kube-api-access-wml27" pod "d135b26d-a1ac-474d-a5ec-147d08187c2d" (UID: "d135b26d-a1ac-474d-a5ec-147d08187c2d"). InnerVolumeSpecName "kube-api-access-wml27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:20.280133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.280102 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5796532d-abfd-4afb-8328-7f3abd2dc06b-kube-api-access-lw8tf" pod "5796532d-abfd-4afb-8328-7f3abd2dc06b" (UID: "5796532d-abfd-4afb-8328-7f3abd2dc06b"). InnerVolumeSpecName "kube-api-access-lw8tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:20.378172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.378133 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5796532d-abfd-4afb-8328-7f3abd2dc06b-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:20.378172 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.378169 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8tf\" (UniqueName: \"kubernetes.io/projected/5796532d-abfd-4afb-8328-7f3abd2dc06b-kube-api-access-lw8tf\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:20.378423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.378182 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d135b26d-a1ac-474d-a5ec-147d08187c2d-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:20.378423 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.378196 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wml27\" (UniqueName: \"kubernetes.io/projected/d135b26d-a1ac-474d-a5ec-147d08187c2d-kube-api-access-wml27\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:20.991433 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.991382 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" event={"ID":"5796532d-abfd-4afb-8328-7f3abd2dc06b","Type":"ContainerDied","Data":"bf5fedc6a1903b0ed83cd0f7fae3ad8258c58ba65f7327be2a68df7b7b802289"} Oct 31 21:45:20.991970 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.991501 5188 scope.go:122] "RemoveContainer" containerID="3a87c062850d730e9b73723bff351fb305e7f6f32db4be587dcd9676959dcb9a" Oct 31 21:45:20.991970 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.991409 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp" Oct 31 21:45:20.998685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.998649 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" event={"ID":"d135b26d-a1ac-474d-a5ec-147d08187c2d","Type":"ContainerDied","Data":"dff7bcebaca2e16c45f5aefeb660e3187e67a7ffd247239735edd3f06ad48073"} Oct 31 21:45:20.998831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:20.998782 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4" Oct 31 21:45:21.010229 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.009491 5188 scope.go:122] "RemoveContainer" containerID="23bb35a2f0a5c5e82443339a1a8b7ce236701e3158e273bf5dfe6684d2ca6cd6" Oct 31 21:45:21.047874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.047841 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb"] Oct 31 21:45:21.106764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.106525 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp"] Oct 31 21:45:21.112318 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.112283 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f4569e-29891385-bvsqp"] Oct 31 21:45:21.185864 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.185097 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:21.185864 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.185097 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4"] Oct 31 21:45:21.189957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.189923 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891385-tdps4"] Oct 31 21:45:21.291927 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.291895 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-kube-api-access-7brtl\" (UniqueName: \"kubernetes.io/projected/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-kube-api-access-7brtl\") pod \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\" (UID: \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\") " Oct 31 21:45:21.292112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.292002 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-maas-api-service-ca\") pod \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\" (UID: \"4e3a68d7-cae6-425e-9cd9-9abf96df76b9\") " Oct 31 21:45:21.292453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.292425 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-maas-api-service-ca" pod "4e3a68d7-cae6-425e-9cd9-9abf96df76b9" (UID: "4e3a68d7-cae6-425e-9cd9-9abf96df76b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:21.292638 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.292621 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:21.294937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.294903 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-kube-api-access-7brtl" pod "4e3a68d7-cae6-425e-9cd9-9abf96df76b9" (UID: "4e3a68d7-cae6-425e-9cd9-9abf96df76b9"). InnerVolumeSpecName "kube-api-access-7brtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:21.311119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.311019 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:21.394033 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.393997 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c436acde-98b5-49b9-afa4-efd29ef06c48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c436acde-98b5-49b9-afa4-efd29ef06c48-maas-api-service-ca\") pod \"c436acde-98b5-49b9-afa4-efd29ef06c48\" (UID: \"c436acde-98b5-49b9-afa4-efd29ef06c48\") " Oct 31 21:45:21.394263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.394136 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c436acde-98b5-49b9-afa4-efd29ef06c48-kube-api-access-nh6ln\" (UniqueName: \"kubernetes.io/projected/c436acde-98b5-49b9-afa4-efd29ef06c48-kube-api-access-nh6ln\") pod \"c436acde-98b5-49b9-afa4-efd29ef06c48\" (UID: \"c436acde-98b5-49b9-afa4-efd29ef06c48\") " Oct 31 21:45:21.394453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.394420 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c436acde-98b5-49b9-afa4-efd29ef06c48-maas-api-service-ca" pod "c436acde-98b5-49b9-afa4-efd29ef06c48" (UID: "c436acde-98b5-49b9-afa4-efd29ef06c48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:21.394587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.394565 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c436acde-98b5-49b9-afa4-efd29ef06c48-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:21.394647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.394594 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7brtl\" (UniqueName: \"kubernetes.io/projected/4e3a68d7-cae6-425e-9cd9-9abf96df76b9-kube-api-access-7brtl\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:21.396495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.396468 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c436acde-98b5-49b9-afa4-efd29ef06c48-kube-api-access-nh6ln" pod "c436acde-98b5-49b9-afa4-efd29ef06c48" (UID: "c436acde-98b5-49b9-afa4-efd29ef06c48"). InnerVolumeSpecName "kube-api-access-nh6ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:21.474820 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.474777 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" path="/var/lib/kubelet/pods/5796532d-abfd-4afb-8328-7f3abd2dc06b/volumes" Oct 31 21:45:21.475245 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.475230 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" path="/var/lib/kubelet/pods/a227657d-f456-40a6-a2ad-9595c762d4b1/volumes" Oct 31 21:45:21.475669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.475653 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" path="/var/lib/kubelet/pods/d135b26d-a1ac-474d-a5ec-147d08187c2d/volumes" Oct 31 21:45:21.496158 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:21.496099 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh6ln\" (UniqueName: \"kubernetes.io/projected/c436acde-98b5-49b9-afa4-efd29ef06c48-kube-api-access-nh6ln\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:22.011726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.011692 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" Oct 31 21:45:22.011726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.011706 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb" event={"ID":"c436acde-98b5-49b9-afa4-efd29ef06c48","Type":"ContainerDied","Data":"0cbf043d228d63356d73e5aa14322b8da1dbb0d36190d8549b1b1113967aaffc"} Oct 31 21:45:22.012287 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.011789 5188 scope.go:122] "RemoveContainer" containerID="7db3cc060091677f48a254af01acb1ae5347298efd8b35becfadd142509853a2" Oct 31 21:45:22.015291 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.015252 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" Oct 31 21:45:22.015498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.015301 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428" event={"ID":"4e3a68d7-cae6-425e-9cd9-9abf96df76b9","Type":"ContainerDied","Data":"163d1cecd0e27e3cd33f67da434ebbc40f3b85accfd040035562d82e309fcece"} Oct 31 21:45:22.024697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.024673 5188 scope.go:122] "RemoveContainer" containerID="005c458be536400b66d348962679c01fde79dcd1872ab59017cf9fb388fef148" Oct 31 21:45:22.060118 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.060084 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb"] Oct 31 21:45:22.069369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.069334 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891385-bp6qb"] Oct 31 21:45:22.104382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.104308 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428"] Oct 31 21:45:22.107976 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:22.107940 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891385-zb428"] Oct 31 21:45:23.479255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:23.479211 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" path="/var/lib/kubelet/pods/4e3a68d7-cae6-425e-9cd9-9abf96df76b9/volumes" Oct 31 21:45:23.479978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:23.479950 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" path="/var/lib/kubelet/pods/c436acde-98b5-49b9-afa4-efd29ef06c48/volumes" Oct 31 21:45:30.870765 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:30.870701 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v"] Oct 31 21:45:30.871358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:30.871006 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" podUID="bd05979e-2766-4060-bb1b-38eeb0faf012" containerName="maas-api" containerID="cri-o://ed9d05e143e92120eb6a463d5855260c1f599c8edcaf3733ccdeadfa0b10400e" gracePeriod=30 Oct 31 21:45:30.967505 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:30.967466 5188 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd05979e_2766_4060_bb1b_38eeb0faf012.slice/crio-conmon-ed9d05e143e92120eb6a463d5855260c1f599c8edcaf3733ccdeadfa0b10400e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd05979e_2766_4060_bb1b_38eeb0faf012.slice/crio-ed9d05e143e92120eb6a463d5855260c1f599c8edcaf3733ccdeadfa0b10400e.scope\": RecentStats: unable to find data in memory cache]" Oct 31 21:45:31.040344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.040289 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474"] Oct 31 21:45:31.040619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.040587 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" podUID="494ded1a-7e61-490c-a73b-367af7969fe0" containerName="payload-processing" containerID="cri-o://b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6" gracePeriod=30 Oct 31 21:45:31.082769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.082727 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6"] Oct 31 21:45:31.083058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.083024 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" podUID="9a257d42-2fe2-4027-b606-5dd0a86b1d2c" containerName="payload-pre-processing" containerID="cri-o://1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0" gracePeriod=30 Oct 31 21:45:31.098533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.098496 5188 generic.go:356] "Generic (PLEG): container finished" podID="bd05979e-2766-4060-bb1b-38eeb0faf012" containerID="ed9d05e143e92120eb6a463d5855260c1f599c8edcaf3733ccdeadfa0b10400e" exitCode=0 Oct 31 21:45:31.098911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.098889 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" event={"ID":"bd05979e-2766-4060-bb1b-38eeb0faf012","Type":"ContainerDied","Data":"ed9d05e143e92120eb6a463d5855260c1f599c8edcaf3733ccdeadfa0b10400e"} Oct 31 21:45:31.240375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.239714 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:45:31.330348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.325970 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-maas-api-tls\") pod \"bd05979e-2766-4060-bb1b-38eeb0faf012\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " Oct 31 21:45:31.330348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.326113 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd05979e-2766-4060-bb1b-38eeb0faf012-kube-api-access-z54mk\" (UniqueName: \"kubernetes.io/projected/bd05979e-2766-4060-bb1b-38eeb0faf012-kube-api-access-z54mk\") pod \"bd05979e-2766-4060-bb1b-38eeb0faf012\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " Oct 31 21:45:31.330348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.326137 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd05979e-2766-4060-bb1b-38eeb0faf012-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd05979e-2766-4060-bb1b-38eeb0faf012-openshift-service-ca\") pod \"bd05979e-2766-4060-bb1b-38eeb0faf012\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " Oct 31 21:45:31.330348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.326181 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-metrics-certs\") pod \"bd05979e-2766-4060-bb1b-38eeb0faf012\" (UID: \"bd05979e-2766-4060-bb1b-38eeb0faf012\") " Oct 31 21:45:31.330348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.327216 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd05979e-2766-4060-bb1b-38eeb0faf012-openshift-service-ca" pod "bd05979e-2766-4060-bb1b-38eeb0faf012" (UID: "bd05979e-2766-4060-bb1b-38eeb0faf012"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:31.330348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.329525 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-maas-api-tls" pod "bd05979e-2766-4060-bb1b-38eeb0faf012" (UID: "bd05979e-2766-4060-bb1b-38eeb0faf012"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:31.330847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.330439 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd05979e-2766-4060-bb1b-38eeb0faf012-kube-api-access-z54mk" pod "bd05979e-2766-4060-bb1b-38eeb0faf012" (UID: "bd05979e-2766-4060-bb1b-38eeb0faf012"). InnerVolumeSpecName "kube-api-access-z54mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:31.333025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.332976 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-metrics-certs" pod "bd05979e-2766-4060-bb1b-38eeb0faf012" (UID: "bd05979e-2766-4060-bb1b-38eeb0faf012"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:31.407307 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.407269 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:45:31.428450 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.428413 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54mk\" (UniqueName: \"kubernetes.io/projected/bd05979e-2766-4060-bb1b-38eeb0faf012-kube-api-access-z54mk\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:31.428450 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.428450 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd05979e-2766-4060-bb1b-38eeb0faf012-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:31.428729 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.428469 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:31.428729 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.428483 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd05979e-2766-4060-bb1b-38eeb0faf012-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:31.451258 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.451222 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:45:31.530859 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.529364 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/494ded1a-7e61-490c-a73b-367af7969fe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494ded1a-7e61-490c-a73b-367af7969fe0-plugins-config-volume\") pod \"494ded1a-7e61-490c-a73b-367af7969fe0\" (UID: \"494ded1a-7e61-490c-a73b-367af7969fe0\") " Oct 31 21:45:31.530859 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.530018 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494ded1a-7e61-490c-a73b-367af7969fe0-plugins-config-volume" pod "494ded1a-7e61-490c-a73b-367af7969fe0" (UID: "494ded1a-7e61-490c-a73b-367af7969fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:31.530859 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.530185 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-kube-api-access-qhqvw\" (UniqueName: \"kubernetes.io/projected/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-kube-api-access-qhqvw\") pod \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\" (UID: \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\") " Oct 31 21:45:31.530859 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.530823 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-plugins-config-volume\") pod \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\" (UID: \"9a257d42-2fe2-4027-b606-5dd0a86b1d2c\") " Oct 31 21:45:31.530859 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.530862 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/494ded1a-7e61-490c-a73b-367af7969fe0-kube-api-access-6khvb\" (UniqueName: \"kubernetes.io/projected/494ded1a-7e61-490c-a73b-367af7969fe0-kube-api-access-6khvb\") pod \"494ded1a-7e61-490c-a73b-367af7969fe0\" (UID: \"494ded1a-7e61-490c-a73b-367af7969fe0\") " Oct 31 21:45:31.531479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.531441 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/494ded1a-7e61-490c-a73b-367af7969fe0-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:31.549683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.533058 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-plugins-config-volume" pod "9a257d42-2fe2-4027-b606-5dd0a86b1d2c" (UID: "9a257d42-2fe2-4027-b606-5dd0a86b1d2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:31.555417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.555347 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494ded1a-7e61-490c-a73b-367af7969fe0-kube-api-access-6khvb" pod "494ded1a-7e61-490c-a73b-367af7969fe0" (UID: "494ded1a-7e61-490c-a73b-367af7969fe0"). InnerVolumeSpecName "kube-api-access-6khvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:31.558946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.558912 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-kube-api-access-qhqvw" pod "9a257d42-2fe2-4027-b606-5dd0a86b1d2c" (UID: "9a257d42-2fe2-4027-b606-5dd0a86b1d2c"). InnerVolumeSpecName "kube-api-access-qhqvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:31.559966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.559876 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78"] Oct 31 21:45:31.560779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560760 5188 cpu_manager.go:414] "containerMap: removing container" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerName="cleanup" Oct 31 21:45:31.560851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560780 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerName="cleanup" Oct 31 21:45:31.560851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560791 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerName="cleanup" Oct 31 21:45:31.560851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560796 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerName="cleanup" Oct 31 21:45:31.560851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560823 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerName="cleanup" Oct 31 21:45:31.560851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560831 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerName="cleanup" Oct 31 21:45:31.560851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560837 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerName="cleanup" Oct 31 21:45:31.560851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560843 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerName="cleanup" Oct 31 21:45:31.560851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560854 5188 cpu_manager.go:414] "containerMap: removing container" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560858 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560864 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560868 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560882 5188 cpu_manager.go:414] "containerMap: removing container" podUID="a4a78bb6-6094-454f-915a-a76064ab3b22" containerName="payload-pre-processing" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560887 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a78bb6-6094-454f-915a-a76064ab3b22" containerName="payload-pre-processing" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560899 5188 cpu_manager.go:414] "containerMap: removing container" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560906 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560913 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560918 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560931 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e168f19b-bf27-4c7c-90a5-a9080d8dbc4d" containerName="payload-processing" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560935 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e168f19b-bf27-4c7c-90a5-a9080d8dbc4d" containerName="payload-processing" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560943 5188 cpu_manager.go:414] "containerMap: removing container" podUID="494ded1a-7e61-490c-a73b-367af7969fe0" containerName="payload-processing" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560948 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="494ded1a-7e61-490c-a73b-367af7969fe0" containerName="payload-processing" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560955 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560959 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560967 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560973 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560982 5188 cpu_manager.go:414] "containerMap: removing container" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560987 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.560995 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9a257d42-2fe2-4027-b606-5dd0a86b1d2c" containerName="payload-pre-processing" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561000 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a257d42-2fe2-4027-b606-5dd0a86b1d2c" containerName="payload-pre-processing" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561007 5188 cpu_manager.go:414] "containerMap: removing container" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561012 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerName="cleanup" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561020 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bd05979e-2766-4060-bb1b-38eeb0faf012" containerName="maas-api" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561025 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd05979e-2766-4060-bb1b-38eeb0faf012" containerName="maas-api" Oct 31 21:45:31.561110 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561116 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561126 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e168f19b-bf27-4c7c-90a5-a9080d8dbc4d" containerName="payload-processing" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561138 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="494ded1a-7e61-490c-a73b-367af7969fe0" containerName="payload-processing" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561145 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561151 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561156 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561162 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a257d42-2fe2-4027-b606-5dd0a86b1d2c" containerName="payload-pre-processing" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561168 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561173 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd05979e-2766-4060-bb1b-38eeb0faf012" containerName="maas-api" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561179 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561185 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561191 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561198 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561205 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerName="cleanup" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561211 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a78bb6-6094-454f-915a-a76064ab3b22" containerName="payload-pre-processing" Oct 31 21:45:31.562126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.561217 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerName="cleanup" Oct 31 21:45:31.565274 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.565252 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.584651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.584611 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78"] Oct 31 21:45:31.650756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.650671 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.650756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.650726 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k546t\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-kube-api-access-k546t\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.650980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.650801 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.650980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.650853 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.650980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.650889 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a58da4-2103-41cf-9493-604d51c5723c-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.651094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.650980 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.651094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.651044 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.651094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.651066 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a58da4-2103-41cf-9493-604d51c5723c-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.651209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.651108 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.651209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.651154 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:31.651209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.651165 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6khvb\" (UniqueName: \"kubernetes.io/projected/494ded1a-7e61-490c-a73b-367af7969fe0-kube-api-access-6khvb\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:31.651209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.651176 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhqvw\" (UniqueName: \"kubernetes.io/projected/9a257d42-2fe2-4027-b606-5dd0a86b1d2c-kube-api-access-qhqvw\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:45:31.751898 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.751851 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k546t\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-kube-api-access-k546t\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.751898 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.751896 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.751927 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.751955 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a58da4-2103-41cf-9493-604d51c5723c-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752018 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752115 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752139 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a58da4-2103-41cf-9493-604d51c5723c-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752179 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752235 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752346 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752474 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752603 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752646 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.752868 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.752846 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a58da4-2103-41cf-9493-604d51c5723c-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.754556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.754533 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.754703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.754683 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a58da4-2103-41cf-9493-604d51c5723c-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.773899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.773860 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.774058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.773923 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k546t\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-kube-api-access-k546t\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:31.906673 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:31.906571 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:32.108413 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.108369 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" event={"ID":"bd05979e-2766-4060-bb1b-38eeb0faf012","Type":"ContainerDied","Data":"3bc681a8b9ffce8f85bc94e2db1a3f2544d21db53b86ba4766eb6dbae5c161e4"} Oct 31 21:45:32.108413 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.108400 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v" Oct 31 21:45:32.108748 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.108459 5188 scope.go:122] "RemoveContainer" containerID="ed9d05e143e92120eb6a463d5855260c1f599c8edcaf3733ccdeadfa0b10400e" Oct 31 21:45:32.111442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.110553 5188 generic.go:356] "Generic (PLEG): container finished" podID="9a257d42-2fe2-4027-b606-5dd0a86b1d2c" containerID="1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0" exitCode=0 Oct 31 21:45:32.111442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.110686 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" event={"ID":"9a257d42-2fe2-4027-b606-5dd0a86b1d2c","Type":"ContainerDied","Data":"1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0"} Oct 31 21:45:32.111442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.110733 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" event={"ID":"9a257d42-2fe2-4027-b606-5dd0a86b1d2c","Type":"ContainerDied","Data":"1ba48d318936c86288b5f1dfe519d061b9f000bc822d1610e302546e4e868349"} Oct 31 21:45:32.111442 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.110811 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6" Oct 31 21:45:32.113730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.113471 5188 generic.go:356] "Generic (PLEG): container finished" podID="494ded1a-7e61-490c-a73b-367af7969fe0" containerID="b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6" exitCode=0 Oct 31 21:45:32.113730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.113507 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" event={"ID":"494ded1a-7e61-490c-a73b-367af7969fe0","Type":"ContainerDied","Data":"b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6"} Oct 31 21:45:32.113730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.113561 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" event={"ID":"494ded1a-7e61-490c-a73b-367af7969fe0","Type":"ContainerDied","Data":"b4d6a070ab7c05b70c1e01ee94831f73e6bbb81fbf99c0b8dfb07ae2e9feda43"} Oct 31 21:45:32.113730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.113518 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474" Oct 31 21:45:32.122743 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.122719 5188 scope.go:122] "RemoveContainer" containerID="1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0" Oct 31 21:45:32.135619 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.135596 5188 scope.go:122] "RemoveContainer" containerID="1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0" Oct 31 21:45:32.135993 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:32.135964 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0\": container with ID starting with 1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0 not found: ID does not exist" containerID="1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0" Oct 31 21:45:32.136119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.135997 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0"} err="failed to get container status \"1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0\": rpc error: code = NotFound desc = could not find container \"1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0\": container with ID starting with 1383ce175972652b197c934a5bdc231a67ffe8263c99243e62eec9eed9ccdec0 not found: ID does not exist" Oct 31 21:45:32.136119 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.136016 5188 scope.go:122] "RemoveContainer" containerID="b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6" Oct 31 21:45:32.147000 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.146979 5188 scope.go:122] "RemoveContainer" containerID="b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6" Oct 31 21:45:32.147381 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:32.147348 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6\": container with ID starting with b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6 not found: ID does not exist" containerID="b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6" Oct 31 21:45:32.147498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.147391 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6"} err="failed to get container status \"b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6\": rpc error: code = NotFound desc = could not find container \"b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6\": container with ID starting with b2de112d59b093a08a280f77163812e905078ff847deffb72888ab517dbcd3d6 not found: ID does not exist" Oct 31 21:45:32.168595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.168522 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v"] Oct 31 21:45:32.174652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.174611 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d43b66-649469d896-qxg7v"] Oct 31 21:45:32.214697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.214661 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474"] Oct 31 21:45:32.220741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.220710 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d43b66-64664b7648-xk474"] Oct 31 21:45:32.249805 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.249770 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6"] Oct 31 21:45:32.253704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.253674 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d43b66-85b7687f95-2fgh6"] Oct 31 21:45:32.281295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.281264 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78"] Oct 31 21:45:32.283315 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:45:32.283287 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a58da4_2103_41cf_9493_604d51c5723c.slice/crio-9342697c4c03dd277e2e7debade78e185b9a9f55784fe90ad6c902038d575f2e WatchSource:0}: Error finding container 9342697c4c03dd277e2e7debade78e185b9a9f55784fe90ad6c902038d575f2e: Status 404 returned error can't find the container with id 9342697c4c03dd277e2e7debade78e185b9a9f55784fe90ad6c902038d575f2e Oct 31 21:45:32.286871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.286824 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:45:32.286993 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.286919 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:45:32.286993 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:32.286965 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:45:32.538957 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:32.538874 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:45:33.128412 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:33.128372 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" event={"ID":"45a58da4-2103-41cf-9493-604d51c5723c","Type":"ContainerStarted","Data":"2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c"} Oct 31 21:45:33.128810 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:33.128464 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" event={"ID":"45a58da4-2103-41cf-9493-604d51c5723c","Type":"ContainerStarted","Data":"9342697c4c03dd277e2e7debade78e185b9a9f55784fe90ad6c902038d575f2e"} Oct 31 21:45:33.193616 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:33.193549 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" podStartSLOduration=2.193528857 podStartE2EDuration="2.193528857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:33.178476726 +0000 UTC m=+1779.871434120" watchObservedRunningTime="2026-10-31 21:45:33.193528857 +0000 UTC m=+1779.886486252" Oct 31 21:45:33.474800 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:33.474714 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="494ded1a-7e61-490c-a73b-367af7969fe0" path="/var/lib/kubelet/pods/494ded1a-7e61-490c-a73b-367af7969fe0/volumes" Oct 31 21:45:33.475706 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:33.475684 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a257d42-2fe2-4027-b606-5dd0a86b1d2c" path="/var/lib/kubelet/pods/9a257d42-2fe2-4027-b606-5dd0a86b1d2c/volumes" Oct 31 21:45:33.476449 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:33.476431 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd05979e-2766-4060-bb1b-38eeb0faf012" path="/var/lib/kubelet/pods/bd05979e-2766-4060-bb1b-38eeb0faf012/volumes" Oct 31 21:45:33.908081 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:33.908044 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:33.909802 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:33.909772 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 21:45:33.909964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:33.909823 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 21:45:34.908850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:34.908813 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 21:45:34.909418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:34.908872 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 21:45:35.908806 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:35.908755 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 21:45:35.909010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:35.908851 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 21:45:36.907759 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:36.907718 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 21:45:36.907981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:36.907790 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 21:45:37.098350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.097021 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq"] Oct 31 21:45:37.104374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104167 5188 cpu_manager.go:414] "containerMap: removing container" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerName="cleanup" Oct 31 21:45:37.104374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104205 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerName="cleanup" Oct 31 21:45:37.104374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104240 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerName="cleanup" Oct 31 21:45:37.104374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104249 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c436acde-98b5-49b9-afa4-efd29ef06c48" containerName="cleanup" Oct 31 21:45:37.104374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104262 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerName="cleanup" Oct 31 21:45:37.104374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104271 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerName="cleanup" Oct 31 21:45:37.104895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104428 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="a227657d-f456-40a6-a2ad-9595c762d4b1" containerName="cleanup" Oct 31 21:45:37.104895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104449 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d135b26d-a1ac-474d-a5ec-147d08187c2d" containerName="cleanup" Oct 31 21:45:37.104895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104459 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3a68d7-cae6-425e-9cd9-9abf96df76b9" containerName="cleanup" Oct 31 21:45:37.104895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.104472 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5796532d-abfd-4afb-8328-7f3abd2dc06b" containerName="cleanup" Oct 31 21:45:37.110098 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.110064 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.127132 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.126945 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-1f523a-serving-cert\"" Oct 31 21:45:37.127764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.127659 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq"] Oct 31 21:45:37.140581 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.140541 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.140737 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.140663 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8427cb0a-6554-4dfb-8779-647eac588f68-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.140786 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.140769 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-metrics-certs\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.140863 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.140841 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm6lj\" (UniqueName: \"kubernetes.io/projected/8427cb0a-6554-4dfb-8779-647eac588f68-kube-api-access-wm6lj\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.242380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.242221 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-metrics-certs\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.242380 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.242317 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm6lj\" (UniqueName: \"kubernetes.io/projected/8427cb0a-6554-4dfb-8779-647eac588f68-kube-api-access-wm6lj\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.242658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.242438 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.242658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.242546 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8427cb0a-6554-4dfb-8779-647eac588f68-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.242658 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:37.242636 5188 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-serving-cert: secret "maas-api-e2e-praxis-b-1f523a-serving-cert" not found Oct 31 21:45:37.242830 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:45:37.242723 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls podName:8427cb0a-6554-4dfb-8779-647eac588f68 nodeName:}" failed. No retries permitted until 2026-10-31 21:45:37.742700491 +0000 UTC m=+1784.435657885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls") pod "maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" (UID: "8427cb0a-6554-4dfb-8779-647eac588f68") : secret "maas-api-e2e-praxis-b-1f523a-serving-cert" not found Oct 31 21:45:37.243231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.243210 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8427cb0a-6554-4dfb-8779-647eac588f68-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.244959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.244930 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-metrics-certs\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.266297 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.266255 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm6lj\" (UniqueName: \"kubernetes.io/projected/8427cb0a-6554-4dfb-8779-647eac588f68-kube-api-access-wm6lj\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.749760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.749712 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.752232 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.752196 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls\") pod \"maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:37.907985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.907943 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 21:45:37.908206 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:37.908026 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 21:45:38.029783 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:38.029687 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:38.372948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:38.368909 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq"] Oct 31 21:45:38.449680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:38.448857 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:45:38.907468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:38.907429 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body= Oct 31 21:45:38.907669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:38.907499 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" Oct 31 21:45:39.236755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:39.236661 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" event={"ID":"8427cb0a-6554-4dfb-8779-647eac588f68","Type":"ContainerStarted","Data":"0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792"} Oct 31 21:45:39.236755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:39.236718 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" event={"ID":"8427cb0a-6554-4dfb-8779-647eac588f68","Type":"ContainerStarted","Data":"052daf7d1197e5e697dd659a89385700b60cc3b9635446f4832971aa31f6cd66"} Oct 31 21:45:39.236755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:39.236747 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:39.281241 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:39.281169 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" podStartSLOduration=1.7391397259999999 podStartE2EDuration="2.281131438s" totalImagesPullingTime="541.991712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:39.277443621 +0000 UTC m=+1785.970401015" watchObservedRunningTime="2026-10-31 21:45:39.281131438 +0000 UTC m=+1785.974088830" Oct 31 21:45:40.375618 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:40.375569 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:40.376431 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:40.376404 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:40.377242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:40.377221 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:45:45.251543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:45.251505 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:45:46.606026 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.605982 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94"] Oct 31 21:45:46.611173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.611145 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.632497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.632459 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94"] Oct 31 21:45:46.699022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.698977 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.699254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.699040 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdkb7\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-kube-api-access-jdkb7\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.699254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.699172 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.699402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.699295 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.699402 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.699337 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.699504 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.699487 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.699545 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.699520 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.699598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.699551 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.699641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.699611 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.800721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.800684 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdkb7\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-kube-api-access-jdkb7\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.800912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.800741 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.800912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.800805 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.800912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.800828 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.801058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.800935 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.801114 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.801070 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.801177 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.801131 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.801234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.801176 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.801234 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.801217 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.801357 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.801336 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.801642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.801612 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.801787 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.801759 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.801980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.801779 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.802075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.802017 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.803396 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.803375 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.803505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.803487 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.822034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.822001 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.822175 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.822156 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdkb7\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-kube-api-access-jdkb7\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:46.937660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:46.937623 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:47.272131 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:47.272093 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94"] Oct 31 21:45:47.273577 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:45:47.273549 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec53a0f8_ecca_4451_8e0d_deb15db413c5.slice/crio-01cb2fcd2f6b802f51ecd05c180b79ca4d63041b660d9a26b3a29c07d79a0b51 WatchSource:0}: Error finding container 01cb2fcd2f6b802f51ecd05c180b79ca4d63041b660d9a26b3a29c07d79a0b51: Status 404 returned error can't find the container with id 01cb2fcd2f6b802f51ecd05c180b79ca4d63041b660d9a26b3a29c07d79a0b51 Oct 31 21:45:47.277007 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:47.276967 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:45:47.277097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:47.277048 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:45:47.277097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:47.277079 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:45:47.305186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:47.305149 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" event={"ID":"ec53a0f8-ecca-4451-8e0d-deb15db413c5","Type":"ContainerStarted","Data":"01cb2fcd2f6b802f51ecd05c180b79ca4d63041b660d9a26b3a29c07d79a0b51"} Oct 31 21:45:48.318097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:48.318052 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" event={"ID":"ec53a0f8-ecca-4451-8e0d-deb15db413c5","Type":"ContainerStarted","Data":"406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907"} Oct 31 21:45:48.390350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:48.390253 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" podStartSLOduration=2.39022936 podStartE2EDuration="2.39022936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:48.369276582 +0000 UTC m=+1795.062233972" watchObservedRunningTime="2026-10-31 21:45:48.39022936 +0000 UTC m=+1795.083186753" Oct 31 21:45:48.938527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:48.938489 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:48.940143 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:48.940118 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 21:45:48.940276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:48.940174 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 21:45:49.938832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:49.938798 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 21:45:49.939276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:49.938856 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 21:45:50.938277 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:50.938232 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 21:45:50.938574 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:50.938376 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 21:45:51.938228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:51.938181 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 21:45:51.938693 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:51.938250 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 21:45:52.938717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:52.938671 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 21:45:52.939133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:52.938734 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 21:45:53.851333 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:53.851299 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:45:53.853219 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:53.853193 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:45:53.938877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:53.938839 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body= Oct 31 21:45:53.939419 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:53.938918 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" Oct 31 21:45:55.348214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:55.348156 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:55.349761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:55.349594 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:45:55.350705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:45:55.350680 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:46:02.539384 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:46:02.539315 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:46:12.174192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.174154 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn"] Oct 31 21:46:12.174719 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.174482 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" podUID="7145f155-06c6-4262-a54e-ed2143c7640d" containerName="maas-api" containerID="cri-o://07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67" gracePeriod=30 Oct 31 21:46:12.191259 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.189873 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" podUID="7145f155-06c6-4262-a54e-ed2143c7640d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.203:8443/health\": dial tcp 10.132.0.203:8443: connect: connection refused" Oct 31 21:46:12.346148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.346110 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c"] Oct 31 21:46:12.346422 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.346391 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" podUID="dfdfc871-39b0-46e5-9a2b-62c08b7e2c19" containerName="payload-processing" containerID="cri-o://ca15467d8e793b4e8f85cb069e6b0ccb9c69b1c70cd11a84e24e59155ff84622" gracePeriod=30 Oct 31 21:46:12.389847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.389807 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp"] Oct 31 21:46:12.390138 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.390082 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" podUID="fa020c9c-ebcb-43e9-ab6a-ac14229b7717" containerName="payload-pre-processing" containerID="cri-o://4ec8871675df4c95e76acd73f139f9509d476471cd385b6fc04e91b4c239d592" gracePeriod=30 Oct 31 21:46:12.531394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.528376 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:46:12.568300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.567699 5188 generic.go:356] "Generic (PLEG): container finished" podID="7145f155-06c6-4262-a54e-ed2143c7640d" containerID="07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67" exitCode=0 Oct 31 21:46:12.568300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.567726 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" event={"ID":"7145f155-06c6-4262-a54e-ed2143c7640d","Type":"ContainerDied","Data":"07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67"} Oct 31 21:46:12.568300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.567788 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" event={"ID":"7145f155-06c6-4262-a54e-ed2143c7640d","Type":"ContainerDied","Data":"5c9b404b027f88b204bfc7007c0710dfee154b34951ee2a70e7bafdd33000d6d"} Oct 31 21:46:12.568300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.567815 5188 scope.go:122] "RemoveContainer" containerID="07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67" Oct 31 21:46:12.568300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.568016 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn" Oct 31 21:46:12.580601 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.570962 5188 generic.go:356] "Generic (PLEG): container finished" podID="fa020c9c-ebcb-43e9-ab6a-ac14229b7717" containerID="4ec8871675df4c95e76acd73f139f9509d476471cd385b6fc04e91b4c239d592" exitCode=0 Oct 31 21:46:12.580601 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.571053 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" event={"ID":"fa020c9c-ebcb-43e9-ab6a-ac14229b7717","Type":"ContainerDied","Data":"4ec8871675df4c95e76acd73f139f9509d476471cd385b6fc04e91b4c239d592"} Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.586975 5188 scope.go:122] "RemoveContainer" containerID="07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67" Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.592940 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7145f155-06c6-4262-a54e-ed2143c7640d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7145f155-06c6-4262-a54e-ed2143c7640d-openshift-service-ca\") pod \"7145f155-06c6-4262-a54e-ed2143c7640d\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.593013 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-maas-api-tls\") pod \"7145f155-06c6-4262-a54e-ed2143c7640d\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.593041 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-metrics-certs\") pod \"7145f155-06c6-4262-a54e-ed2143c7640d\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.594041 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7145f155-06c6-4262-a54e-ed2143c7640d-kube-api-access-6rk6z\" (UniqueName: \"kubernetes.io/projected/7145f155-06c6-4262-a54e-ed2143c7640d-kube-api-access-6rk6z\") pod \"7145f155-06c6-4262-a54e-ed2143c7640d\" (UID: \"7145f155-06c6-4262-a54e-ed2143c7640d\") " Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:46:12.595866 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67\": container with ID starting with 07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67 not found: ID does not exist" containerID="07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67" Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.595908 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67"} err="failed to get container status \"07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67\": rpc error: code = NotFound desc = could not find container \"07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67\": container with ID starting with 07f0b01f771b550a5ccffaa2996980a1e9d3e2c7ff254c356929455b6e3b6f67 not found: ID does not exist" Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.597783 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7145f155-06c6-4262-a54e-ed2143c7640d-openshift-service-ca" pod "7145f155-06c6-4262-a54e-ed2143c7640d" (UID: "7145f155-06c6-4262-a54e-ed2143c7640d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.597824 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7145f155-06c6-4262-a54e-ed2143c7640d-kube-api-access-6rk6z" pod "7145f155-06c6-4262-a54e-ed2143c7640d" (UID: "7145f155-06c6-4262-a54e-ed2143c7640d"). InnerVolumeSpecName "kube-api-access-6rk6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.598673 5188 generic.go:356] "Generic (PLEG): container finished" podID="dfdfc871-39b0-46e5-9a2b-62c08b7e2c19" containerID="ca15467d8e793b4e8f85cb069e6b0ccb9c69b1c70cd11a84e24e59155ff84622" exitCode=0 Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.598771 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" event={"ID":"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19","Type":"ContainerDied","Data":"ca15467d8e793b4e8f85cb069e6b0ccb9c69b1c70cd11a84e24e59155ff84622"} Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.600148 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-maas-api-tls" pod "7145f155-06c6-4262-a54e-ed2143c7640d" (UID: "7145f155-06c6-4262-a54e-ed2143c7640d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:12.618201 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.607084 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-metrics-certs" pod "7145f155-06c6-4262-a54e-ed2143c7640d" (UID: "7145f155-06c6-4262-a54e-ed2143c7640d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:12.708990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.708872 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7145f155-06c6-4262-a54e-ed2143c7640d-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:12.708990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.708923 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:12.708990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.708939 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7145f155-06c6-4262-a54e-ed2143c7640d-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:12.708990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.708954 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rk6z\" (UniqueName: \"kubernetes.io/projected/7145f155-06c6-4262-a54e-ed2143c7640d-kube-api-access-6rk6z\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:12.719847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.719467 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:46:12.810220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.809911 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-kube-api-access-hj56k\" (UniqueName: \"kubernetes.io/projected/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-kube-api-access-hj56k\") pod \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\" (UID: \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\") " Oct 31 21:46:12.810220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.809988 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-plugins-config-volume\") pod \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\" (UID: \"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19\") " Oct 31 21:46:12.811503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.811450 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-plugins-config-volume" pod "dfdfc871-39b0-46e5-9a2b-62c08b7e2c19" (UID: "dfdfc871-39b0-46e5-9a2b-62c08b7e2c19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:12.819358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.819283 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-kube-api-access-hj56k" pod "dfdfc871-39b0-46e5-9a2b-62c08b7e2c19" (UID: "dfdfc871-39b0-46e5-9a2b-62c08b7e2c19"). InnerVolumeSpecName "kube-api-access-hj56k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:12.874396 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.868994 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:46:12.912316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.912020 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-plugins-config-volume\") pod \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\" (UID: \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\") " Oct 31 21:46:12.912832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.912703 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-plugins-config-volume" pod "fa020c9c-ebcb-43e9-ab6a-ac14229b7717" (UID: "fa020c9c-ebcb-43e9-ab6a-ac14229b7717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:12.913481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.913084 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-kube-api-access-rfxtl\" (UniqueName: \"kubernetes.io/projected/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-kube-api-access-rfxtl\") pod \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\" (UID: \"fa020c9c-ebcb-43e9-ab6a-ac14229b7717\") " Oct 31 21:46:12.913631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.913609 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj56k\" (UniqueName: \"kubernetes.io/projected/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-kube-api-access-hj56k\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:12.913698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.913637 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:12.913698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.913651 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:12.915933 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.915909 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-kube-api-access-rfxtl" pod "fa020c9c-ebcb-43e9-ab6a-ac14229b7717" (UID: "fa020c9c-ebcb-43e9-ab6a-ac14229b7717"). InnerVolumeSpecName "kube-api-access-rfxtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:12.944940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.944866 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn"] Oct 31 21:46:12.949861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:12.949823 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-09f39e-6644496cf6-6vwwn"] Oct 31 21:46:13.014950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.014861 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfxtl\" (UniqueName: \"kubernetes.io/projected/fa020c9c-ebcb-43e9-ab6a-ac14229b7717-kube-api-access-rfxtl\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:13.476672 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.476635 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7145f155-06c6-4262-a54e-ed2143c7640d" path="/var/lib/kubelet/pods/7145f155-06c6-4262-a54e-ed2143c7640d/volumes" Oct 31 21:46:13.609302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.609263 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" Oct 31 21:46:13.609553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.609297 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c" event={"ID":"dfdfc871-39b0-46e5-9a2b-62c08b7e2c19","Type":"ContainerDied","Data":"63cec64027018ce13f6ddd04546d6032bb6fe7e2b1ef2943282126a8f6d7f925"} Oct 31 21:46:13.609553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.609403 5188 scope.go:122] "RemoveContainer" containerID="ca15467d8e793b4e8f85cb069e6b0ccb9c69b1c70cd11a84e24e59155ff84622" Oct 31 21:46:13.612802 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.612774 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" event={"ID":"fa020c9c-ebcb-43e9-ab6a-ac14229b7717","Type":"ContainerDied","Data":"2aea8d597db58b010d27b7707047dee7081119b701371b56b9593fe13777133f"} Oct 31 21:46:13.612950 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.612860 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp" Oct 31 21:46:13.626851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.626826 5188 scope.go:122] "RemoveContainer" containerID="4ec8871675df4c95e76acd73f139f9509d476471cd385b6fc04e91b4c239d592" Oct 31 21:46:13.658613 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.658558 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c"] Oct 31 21:46:13.663091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.663057 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-796d44785b-8w85c"] Oct 31 21:46:13.691557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.691520 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp"] Oct 31 21:46:13.697108 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:13.697059 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-5d959cd875-8jrmp"] Oct 31 21:46:15.474604 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:15.474567 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfdfc871-39b0-46e5-9a2b-62c08b7e2c19" path="/var/lib/kubelet/pods/dfdfc871-39b0-46e5-9a2b-62c08b7e2c19/volumes" Oct 31 21:46:15.475102 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:15.474992 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa020c9c-ebcb-43e9-ab6a-ac14229b7717" path="/var/lib/kubelet/pods/fa020c9c-ebcb-43e9-ab6a-ac14229b7717/volumes" Oct 31 21:46:19.271505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.271463 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk"] Oct 31 21:46:19.272082 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.272066 5188 cpu_manager.go:414] "containerMap: removing container" podUID="fa020c9c-ebcb-43e9-ab6a-ac14229b7717" containerName="payload-pre-processing" Oct 31 21:46:19.272133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.272083 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa020c9c-ebcb-43e9-ab6a-ac14229b7717" containerName="payload-pre-processing" Oct 31 21:46:19.272133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.272102 5188 cpu_manager.go:414] "containerMap: removing container" podUID="7145f155-06c6-4262-a54e-ed2143c7640d" containerName="maas-api" Oct 31 21:46:19.272133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.272107 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7145f155-06c6-4262-a54e-ed2143c7640d" containerName="maas-api" Oct 31 21:46:19.272133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.272115 5188 cpu_manager.go:414] "containerMap: removing container" podUID="dfdfc871-39b0-46e5-9a2b-62c08b7e2c19" containerName="payload-processing" Oct 31 21:46:19.272133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.272120 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfdfc871-39b0-46e5-9a2b-62c08b7e2c19" containerName="payload-processing" Oct 31 21:46:19.272300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.272215 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa020c9c-ebcb-43e9-ab6a-ac14229b7717" containerName="payload-pre-processing" Oct 31 21:46:19.272300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.272225 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="7145f155-06c6-4262-a54e-ed2143c7640d" containerName="maas-api" Oct 31 21:46:19.272300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.272231 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfdfc871-39b0-46e5-9a2b-62c08b7e2c19" containerName="payload-processing" Oct 31 21:46:19.277127 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.277092 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.296536 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.296492 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk"] Oct 31 21:46:19.302774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.301149 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:46:19.302774 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.301204 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:46:19.422575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.422522 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a059aa-l88tk\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.422575 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.422574 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsxlk\" (UniqueName: \"kubernetes.io/projected/f653e756-ced4-46ec-9100-b4f81380b05a-kube-api-access-zsxlk\") pod \"maas-api-revoke-sub-68a059aa-l88tk\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.422948 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.422697 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a059aa-l88tk\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.523954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.523857 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a059aa-l88tk\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.523954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.523907 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsxlk\" (UniqueName: \"kubernetes.io/projected/f653e756-ced4-46ec-9100-b4f81380b05a-kube-api-access-zsxlk\") pod \"maas-api-revoke-sub-68a059aa-l88tk\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.524277 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.523974 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a059aa-l88tk\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.524587 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.524563 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a059aa-l88tk\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.526506 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.526481 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a059aa-l88tk\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.545473 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.545434 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsxlk\" (UniqueName: \"kubernetes.io/projected/f653e756-ced4-46ec-9100-b4f81380b05a-kube-api-access-zsxlk\") pod \"maas-api-revoke-sub-68a059aa-l88tk\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.594055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.594012 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:19.901714 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:19.901679 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk"] Oct 31 21:46:19.904647 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:46:19.904604 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf653e756_ced4_46ec_9100_b4f81380b05a.slice/crio-a89c78b311066b5a02fcdc94010f4ef0c83bbca21ef6878a2826a982a3737038 WatchSource:0}: Error finding container a89c78b311066b5a02fcdc94010f4ef0c83bbca21ef6878a2826a982a3737038: Status 404 returned error can't find the container with id a89c78b311066b5a02fcdc94010f4ef0c83bbca21ef6878a2826a982a3737038 Oct 31 21:46:20.694722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:20.694675 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" event={"ID":"f653e756-ced4-46ec-9100-b4f81380b05a","Type":"ContainerStarted","Data":"8b51ef9ad5695274c41d45bc99e49f3a17805f3f47f3308a9ac971304a459dee"} Oct 31 21:46:20.695184 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:20.694751 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" event={"ID":"f653e756-ced4-46ec-9100-b4f81380b05a","Type":"ContainerStarted","Data":"a89c78b311066b5a02fcdc94010f4ef0c83bbca21ef6878a2826a982a3737038"} Oct 31 21:46:20.742595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:20.742528 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" podStartSLOduration=1.742509658 podStartE2EDuration="1.742509658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:20.732038545 +0000 UTC m=+1827.424995935" watchObservedRunningTime="2026-10-31 21:46:20.742509658 +0000 UTC m=+1827.435467052" Oct 31 21:46:21.705434 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.705390 5188 generic.go:356] "Generic (PLEG): container finished" podID="f653e756-ced4-46ec-9100-b4f81380b05a" containerID="8b51ef9ad5695274c41d45bc99e49f3a17805f3f47f3308a9ac971304a459dee" exitCode=0 Oct 31 21:46:21.705434 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.705443 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" event={"ID":"f653e756-ced4-46ec-9100-b4f81380b05a","Type":"ContainerDied","Data":"8b51ef9ad5695274c41d45bc99e49f3a17805f3f47f3308a9ac971304a459dee"} Oct 31 21:46:21.844497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.844427 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg"] Oct 31 21:46:21.848331 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.848302 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.874447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.874405 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg"] Oct 31 21:46:21.895257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.895219 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.895470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.895278 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.895470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.895417 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8828d49-b461-46a8-919e-39731dc06b1f-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.895553 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.895525 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.895597 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.895563 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dcxw\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-kube-api-access-9dcxw\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.895644 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.895603 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8828d49-b461-46a8-919e-39731dc06b1f-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.895687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.895664 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.895730 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.895720 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.895770 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.895741 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.997501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.997296 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8828d49-b461-46a8-919e-39731dc06b1f-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.997501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.997431 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.997501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.997468 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.997501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.997496 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.997549 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.997587 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.997694 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8828d49-b461-46a8-919e-39731dc06b1f-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.997834 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.997905 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcxw\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-kube-api-access-9dcxw\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.998015 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.998033 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.998075 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.998358 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:21.999151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:21.998776 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8828d49-b461-46a8-919e-39731dc06b1f-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:22.000204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.000183 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8828d49-b461-46a8-919e-39731dc06b1f-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:22.000362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.000300 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:22.018827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.018791 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:22.019032 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.019006 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dcxw\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-kube-api-access-9dcxw\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:22.166226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.166184 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:22.501498 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.501457 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg"] Oct 31 21:46:22.503091 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:46:22.503053 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8828d49_b461_46a8_919e_39731dc06b1f.slice/crio-de071b13d46a0a71c38ffd68b4a0921419919522ce8c924e7a652a99f73c022e WatchSource:0}: Error finding container de071b13d46a0a71c38ffd68b4a0921419919522ce8c924e7a652a99f73c022e: Status 404 returned error can't find the container with id de071b13d46a0a71c38ffd68b4a0921419919522ce8c924e7a652a99f73c022e Oct 31 21:46:22.506001 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.505962 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:46:22.506129 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.506034 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:46:22.506129 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.506064 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:46:22.713955 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.713916 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" event={"ID":"a8828d49-b461-46a8-919e-39731dc06b1f","Type":"ContainerStarted","Data":"9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802"} Oct 31 21:46:22.714397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.713980 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" event={"ID":"a8828d49-b461-46a8-919e-39731dc06b1f","Type":"ContainerStarted","Data":"de071b13d46a0a71c38ffd68b4a0921419919522ce8c924e7a652a99f73c022e"} Oct 31 21:46:22.754809 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.754676 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" podStartSLOduration=1.754654878 podStartE2EDuration="1.754654878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:22.754026954 +0000 UTC m=+1829.446984348" watchObservedRunningTime="2026-10-31 21:46:22.754654878 +0000 UTC m=+1829.447612273" Oct 31 21:46:22.868912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.868880 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:22.915700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.915656 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f653e756-ced4-46ec-9100-b4f81380b05a-kube-api-access-zsxlk\" (UniqueName: \"kubernetes.io/projected/f653e756-ced4-46ec-9100-b4f81380b05a-kube-api-access-zsxlk\") pod \"f653e756-ced4-46ec-9100-b4f81380b05a\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " Oct 31 21:46:22.915916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.915885 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-openshift-ca\") pod \"f653e756-ced4-46ec-9100-b4f81380b05a\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " Oct 31 21:46:22.916005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.915977 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-xks-ca\") pod \"f653e756-ced4-46ec-9100-b4f81380b05a\" (UID: \"f653e756-ced4-46ec-9100-b4f81380b05a\") " Oct 31 21:46:22.917225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.917186 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-openshift-ca" pod "f653e756-ced4-46ec-9100-b4f81380b05a" (UID: "f653e756-ced4-46ec-9100-b4f81380b05a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:22.935049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.935013 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f653e756-ced4-46ec-9100-b4f81380b05a-kube-api-access-zsxlk" pod "f653e756-ced4-46ec-9100-b4f81380b05a" (UID: "f653e756-ced4-46ec-9100-b4f81380b05a"). InnerVolumeSpecName "kube-api-access-zsxlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:22.935255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:22.935029 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-xks-ca" pod "f653e756-ced4-46ec-9100-b4f81380b05a" (UID: "f653e756-ced4-46ec-9100-b4f81380b05a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:23.028769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:23.028722 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:23.028769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:23.028759 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f653e756-ced4-46ec-9100-b4f81380b05a-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:23.028769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:23.028772 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsxlk\" (UniqueName: \"kubernetes.io/projected/f653e756-ced4-46ec-9100-b4f81380b05a-kube-api-access-zsxlk\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:23.167209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:23.167161 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:23.169505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:23.169469 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 21:46:23.169702 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:23.169527 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 21:46:23.722901 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:23.722866 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" Oct 31 21:46:23.723428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:23.722978 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk" event={"ID":"f653e756-ced4-46ec-9100-b4f81380b05a","Type":"ContainerDied","Data":"a89c78b311066b5a02fcdc94010f4ef0c83bbca21ef6878a2826a982a3737038"} Oct 31 21:46:23.723428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:23.723015 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a89c78b311066b5a02fcdc94010f4ef0c83bbca21ef6878a2826a982a3737038" Oct 31 21:46:24.166990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:24.166947 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 21:46:24.167183 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:24.167016 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 21:46:25.167088 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:25.167040 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 21:46:25.167552 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:25.167110 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 21:46:26.167550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:26.167507 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 21:46:26.168091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:26.167588 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 21:46:27.167336 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.167284 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 21:46:27.167536 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.167384 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 21:46:27.929739 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.929705 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d"] Oct 31 21:46:27.930313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.930289 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f653e756-ced4-46ec-9100-b4f81380b05a" containerName="revoke-keys" Oct 31 21:46:27.930313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.930307 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f653e756-ced4-46ec-9100-b4f81380b05a" containerName="revoke-keys" Oct 31 21:46:27.930490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.930434 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f653e756-ced4-46ec-9100-b4f81380b05a" containerName="revoke-keys" Oct 31 21:46:27.933718 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.933700 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:27.957870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.957841 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b01ff6-dockercfg-gjr2p\"" Oct 31 21:46:27.958080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.957861 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b01ff6\"" Oct 31 21:46:27.962184 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:27.962154 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d"] Oct 31 21:46:28.048352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.048276 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6800f9ca-3103-4c97-82ae-ff7aaa223df3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d\" (UID: \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:28.049704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.049670 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2886v\" (UniqueName: \"kubernetes.io/projected/6800f9ca-3103-4c97-82ae-ff7aaa223df3-kube-api-access-2886v\") pod \"payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d\" (UID: \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:28.056196 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.056158 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr"] Oct 31 21:46:28.063124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.062978 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:28.084103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.084065 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr"] Oct 31 21:46:28.235540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.235419 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 21:46:28.235741 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.235524 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 21:46:28.236679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.236460 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6800f9ca-3103-4c97-82ae-ff7aaa223df3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d\" (UID: \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:28.236679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.236518 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25vd\" (UniqueName: \"kubernetes.io/projected/9966a67d-a22f-451f-987a-448ca7b01aa9-kube-api-access-k25vd\") pod \"payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr\" (UID: \"9966a67d-a22f-451f-987a-448ca7b01aa9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:28.236679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.236586 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9966a67d-a22f-451f-987a-448ca7b01aa9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr\" (UID: \"9966a67d-a22f-451f-987a-448ca7b01aa9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:28.236679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.236620 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2886v\" (UniqueName: \"kubernetes.io/projected/6800f9ca-3103-4c97-82ae-ff7aaa223df3-kube-api-access-2886v\") pod \"payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d\" (UID: \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:28.238097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.238063 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6800f9ca-3103-4c97-82ae-ff7aaa223df3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d\" (UID: \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:28.260139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.260105 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2886v\" (UniqueName: \"kubernetes.io/projected/6800f9ca-3103-4c97-82ae-ff7aaa223df3-kube-api-access-2886v\") pod \"payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d\" (UID: \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:28.337631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.337588 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25vd\" (UniqueName: \"kubernetes.io/projected/9966a67d-a22f-451f-987a-448ca7b01aa9-kube-api-access-k25vd\") pod \"payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr\" (UID: \"9966a67d-a22f-451f-987a-448ca7b01aa9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:28.337854 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.337667 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9966a67d-a22f-451f-987a-448ca7b01aa9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr\" (UID: \"9966a67d-a22f-451f-987a-448ca7b01aa9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:28.338495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.338468 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9966a67d-a22f-451f-987a-448ca7b01aa9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr\" (UID: \"9966a67d-a22f-451f-987a-448ca7b01aa9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:28.359040 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.359010 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25vd\" (UniqueName: \"kubernetes.io/projected/9966a67d-a22f-451f-987a-448ca7b01aa9-kube-api-access-k25vd\") pod \"payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr\" (UID: \"9966a67d-a22f-451f-987a-448ca7b01aa9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:28.539467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.539378 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:28.555313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.555081 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:28.889610 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:46:28.889577 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9966a67d_a22f_451f_987a_448ca7b01aa9.slice/crio-fb1321781979a4ab896482e185ba84d64aee6e8e9e66257f5f9ad8cf9e2e17f1 WatchSource:0}: Error finding container fb1321781979a4ab896482e185ba84d64aee6e8e9e66257f5f9ad8cf9e2e17f1: Status 404 returned error can't find the container with id fb1321781979a4ab896482e185ba84d64aee6e8e9e66257f5f9ad8cf9e2e17f1 Oct 31 21:46:28.889821 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.889792 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr"] Oct 31 21:46:28.925564 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:46:28.925526 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6800f9ca_3103_4c97_82ae_ff7aaa223df3.slice/crio-991559a2eb434ca3783a83a7eb4f72b334a05b98b40fe980683b7abad17380c8 WatchSource:0}: Error finding container 991559a2eb434ca3783a83a7eb4f72b334a05b98b40fe980683b7abad17380c8: Status 404 returned error can't find the container with id 991559a2eb434ca3783a83a7eb4f72b334a05b98b40fe980683b7abad17380c8 Oct 31 21:46:28.925711 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:28.925640 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d"] Oct 31 21:46:29.167784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:29.167692 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 21:46:29.167784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:29.167765 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 21:46:29.772127 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:29.772079 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" event={"ID":"9966a67d-a22f-451f-987a-448ca7b01aa9","Type":"ContainerStarted","Data":"f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e"} Oct 31 21:46:29.772385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:29.772153 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" event={"ID":"9966a67d-a22f-451f-987a-448ca7b01aa9","Type":"ContainerStarted","Data":"fb1321781979a4ab896482e185ba84d64aee6e8e9e66257f5f9ad8cf9e2e17f1"} Oct 31 21:46:29.774811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:29.774219 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" event={"ID":"6800f9ca-3103-4c97-82ae-ff7aaa223df3","Type":"ContainerStarted","Data":"55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062"} Oct 31 21:46:29.774811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:29.774274 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" event={"ID":"6800f9ca-3103-4c97-82ae-ff7aaa223df3","Type":"ContainerStarted","Data":"991559a2eb434ca3783a83a7eb4f72b334a05b98b40fe980683b7abad17380c8"} Oct 31 21:46:29.774811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:29.774302 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:29.812544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:29.812464 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" podStartSLOduration=1.8124432129999999 podStartE2EDuration="1.812443213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:29.808345432 +0000 UTC m=+1836.501302842" watchObservedRunningTime="2026-10-31 21:46:29.812443213 +0000 UTC m=+1836.505400608" Oct 31 21:46:29.849048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:29.848975 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" podStartSLOduration=2.848955412 podStartE2EDuration="2.848955412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:29.84386371 +0000 UTC m=+1836.536821104" watchObservedRunningTime="2026-10-31 21:46:29.848955412 +0000 UTC m=+1836.541912806" Oct 31 21:46:30.214249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:30.214214 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:30.214845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:30.214630 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:30.215850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:30.215814 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:46:30.783916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:30.783871 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:32.540633 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:46:32.540592 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:46:40.788632 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:40.788593 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:46:41.796650 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:41.796620 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:46:45.082334 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.082283 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd"] Oct 31 21:46:45.086798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.086769 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.090663 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.090629 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.090859 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.090710 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.090929 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.090891 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-metrics-certs\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.091012 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.090989 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw5zw\" (UniqueName: \"kubernetes.io/projected/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-kube-api-access-lw5zw\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.104223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.104190 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b01ff6-serving-cert\"" Oct 31 21:46:45.117831 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.117798 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd"] Oct 31 21:46:45.192549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.192512 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.192777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.192597 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.192777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.192687 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-metrics-certs\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.192777 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.192723 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw5zw\" (UniqueName: \"kubernetes.io/projected/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-kube-api-access-lw5zw\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.192942 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:46:45.192769 5188 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-serving-cert: secret "maas-api-e2e-praxis-a-b01ff6-serving-cert" not found Oct 31 21:46:45.192942 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:46:45.192873 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls podName:6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5 nodeName:}" failed. No retries permitted until 2026-10-31 21:46:45.692849998 +0000 UTC m=+1852.385807383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls") pod "maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" (UID: "6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5") : secret "maas-api-e2e-praxis-a-b01ff6-serving-cert" not found Oct 31 21:46:45.193524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.193504 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.195513 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.195487 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-metrics-certs\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.214686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.214653 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw5zw\" (UniqueName: \"kubernetes.io/projected/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-kube-api-access-lw5zw\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.699793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.699748 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.702293 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.702265 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:45.704183 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:45.704162 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:46.025388 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:46.024715 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd"] Oct 31 21:46:46.026469 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:46:46.026435 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a84b8e6_2b1f_478e_b4cb_f0b070fb3fd5.slice/crio-93b992a312b7e60697b43e1e7a975d87e7f44fd3a72dfd2331ed9e3a03c8c781 WatchSource:0}: Error finding container 93b992a312b7e60697b43e1e7a975d87e7f44fd3a72dfd2331ed9e3a03c8c781: Status 404 returned error can't find the container with id 93b992a312b7e60697b43e1e7a975d87e7f44fd3a72dfd2331ed9e3a03c8c781 Oct 31 21:46:46.979352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:46.979296 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" event={"ID":"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5","Type":"ContainerStarted","Data":"c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef"} Oct 31 21:46:46.979870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:46.979361 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" event={"ID":"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5","Type":"ContainerStarted","Data":"93b992a312b7e60697b43e1e7a975d87e7f44fd3a72dfd2331ed9e3a03c8c781"} Oct 31 21:46:47.021048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:47.020979 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" podStartSLOduration=1.6391425499999999 podStartE2EDuration="2.020960791s" totalImagesPullingTime="381.818241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:47.018798347 +0000 UTC m=+1853.711755742" watchObservedRunningTime="2026-10-31 21:46:47.020960791 +0000 UTC m=+1853.713918187" Oct 31 21:46:47.987838 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:47.987795 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:54.004052 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:54.004013 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:46:55.438274 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:55.438233 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk"] Oct 31 21:46:55.438812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:55.438531 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" podUID="1c2e64cf-ece3-4520-9f7e-0e39d51919f5" containerName="maas-api" containerID="cri-o://563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802" gracePeriod=30 Oct 31 21:46:56.035406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.035362 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:46:56.055082 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.055050 5188 generic.go:356] "Generic (PLEG): container finished" podID="1c2e64cf-ece3-4520-9f7e-0e39d51919f5" containerID="563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802" exitCode=0 Oct 31 21:46:56.055263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.055099 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" Oct 31 21:46:56.055263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.055140 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" event={"ID":"1c2e64cf-ece3-4520-9f7e-0e39d51919f5","Type":"ContainerDied","Data":"563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802"} Oct 31 21:46:56.055263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.055196 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk" event={"ID":"1c2e64cf-ece3-4520-9f7e-0e39d51919f5","Type":"ContainerDied","Data":"b9bbb4b682f2a5f952db1fbd9c072ba3d97669525486e8d02e30e933990de3c9"} Oct 31 21:46:56.055263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.055222 5188 scope.go:122] "RemoveContainer" containerID="563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802" Oct 31 21:46:56.069058 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.069030 5188 scope.go:122] "RemoveContainer" containerID="563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802" Oct 31 21:46:56.069583 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:46:56.069557 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802\": container with ID starting with 563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802 not found: ID does not exist" containerID="563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802" Oct 31 21:46:56.069699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.069592 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802"} err="failed to get container status \"563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802\": rpc error: code = NotFound desc = could not find container \"563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802\": container with ID starting with 563f04721f1a38302ffa2069f0d37ee4906ea6deb2c204674b05b8ea0645c802 not found: ID does not exist" Oct 31 21:46:56.145658 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.145620 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-kube-api-access-wgs9l\" (UniqueName: \"kubernetes.io/projected/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-kube-api-access-wgs9l\") pod \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " Oct 31 21:46:56.145871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.145739 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-metrics-certs\") pod \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " Oct 31 21:46:56.145871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.145847 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-openshift-service-ca\") pod \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " Oct 31 21:46:56.145989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.145917 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-maas-api-tls\") pod \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\" (UID: \"1c2e64cf-ece3-4520-9f7e-0e39d51919f5\") " Oct 31 21:46:56.146240 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.146210 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-openshift-service-ca" pod "1c2e64cf-ece3-4520-9f7e-0e39d51919f5" (UID: "1c2e64cf-ece3-4520-9f7e-0e39d51919f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:56.146600 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.146581 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:56.148026 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.148002 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-metrics-certs" pod "1c2e64cf-ece3-4520-9f7e-0e39d51919f5" (UID: "1c2e64cf-ece3-4520-9f7e-0e39d51919f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:56.148139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.148104 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-maas-api-tls" pod "1c2e64cf-ece3-4520-9f7e-0e39d51919f5" (UID: "1c2e64cf-ece3-4520-9f7e-0e39d51919f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:56.148204 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.148162 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-kube-api-access-wgs9l" pod "1c2e64cf-ece3-4520-9f7e-0e39d51919f5" (UID: "1c2e64cf-ece3-4520-9f7e-0e39d51919f5"). InnerVolumeSpecName "kube-api-access-wgs9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:56.248013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.247904 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:56.248013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.247944 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgs9l\" (UniqueName: \"kubernetes.io/projected/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-kube-api-access-wgs9l\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:56.248013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.247959 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c2e64cf-ece3-4520-9f7e-0e39d51919f5-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:46:56.411372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.403599 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk"] Oct 31 21:46:56.411791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:56.411632 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-80542b-57ff856d68-cfwnk"] Oct 31 21:46:57.474945 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:46:57.474909 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2e64cf-ece3-4520-9f7e-0e39d51919f5" path="/var/lib/kubelet/pods/1c2e64cf-ece3-4520-9f7e-0e39d51919f5/volumes" Oct 31 21:47:02.771690 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:47:02.771631 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:47:03.802620 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:03.802567 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq"] Oct 31 21:47:03.803208 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:03.802977 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" podUID="8427cb0a-6554-4dfb-8779-647eac588f68" containerName="maas-api" containerID="cri-o://0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792" gracePeriod=30 Oct 31 21:47:04.165963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.165931 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:47:04.312387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.305453 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8427cb0a-6554-4dfb-8779-647eac588f68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8427cb0a-6554-4dfb-8779-647eac588f68-openshift-service-ca\") pod \"8427cb0a-6554-4dfb-8779-647eac588f68\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " Oct 31 21:47:04.312387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.305873 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls\") pod \"8427cb0a-6554-4dfb-8779-647eac588f68\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " Oct 31 21:47:04.312387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.306085 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8427cb0a-6554-4dfb-8779-647eac588f68-kube-api-access-wm6lj\" (UniqueName: \"kubernetes.io/projected/8427cb0a-6554-4dfb-8779-647eac588f68-kube-api-access-wm6lj\") pod \"8427cb0a-6554-4dfb-8779-647eac588f68\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " Oct 31 21:47:04.312387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.306526 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-metrics-certs\") pod \"8427cb0a-6554-4dfb-8779-647eac588f68\" (UID: \"8427cb0a-6554-4dfb-8779-647eac588f68\") " Oct 31 21:47:04.312387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.309018 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8427cb0a-6554-4dfb-8779-647eac588f68-openshift-service-ca" pod "8427cb0a-6554-4dfb-8779-647eac588f68" (UID: "8427cb0a-6554-4dfb-8779-647eac588f68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:04.326644 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.326596 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8427cb0a-6554-4dfb-8779-647eac588f68-kube-api-access-wm6lj" pod "8427cb0a-6554-4dfb-8779-647eac588f68" (UID: "8427cb0a-6554-4dfb-8779-647eac588f68"). InnerVolumeSpecName "kube-api-access-wm6lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:04.328103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.328072 5188 generic.go:356] "Generic (PLEG): container finished" podID="8427cb0a-6554-4dfb-8779-647eac588f68" containerID="0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792" exitCode=0 Oct 31 21:47:04.328344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.328303 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" event={"ID":"8427cb0a-6554-4dfb-8779-647eac588f68","Type":"ContainerDied","Data":"0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792"} Oct 31 21:47:04.328463 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.328400 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" event={"ID":"8427cb0a-6554-4dfb-8779-647eac588f68","Type":"ContainerDied","Data":"052daf7d1197e5e697dd659a89385700b60cc3b9635446f4832971aa31f6cd66"} Oct 31 21:47:04.328463 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.328437 5188 scope.go:122] "RemoveContainer" containerID="0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792" Oct 31 21:47:04.328639 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.328621 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq" Oct 31 21:47:04.329790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.329756 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-metrics-certs" pod "8427cb0a-6554-4dfb-8779-647eac588f68" (UID: "8427cb0a-6554-4dfb-8779-647eac588f68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:04.330036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.330011 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls" pod "8427cb0a-6554-4dfb-8779-647eac588f68" (UID: "8427cb0a-6554-4dfb-8779-647eac588f68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:04.363429 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.363405 5188 scope.go:122] "RemoveContainer" containerID="0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792" Oct 31 21:47:04.363788 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:47:04.363759 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792\": container with ID starting with 0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792 not found: ID does not exist" containerID="0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792" Oct 31 21:47:04.363897 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.363793 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792"} err="failed to get container status \"0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792\": rpc error: code = NotFound desc = could not find container \"0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792\": container with ID starting with 0edf29fd2112b395693259364820da4b479c28b0c8d4cd8e6ccf611ef2c77792 not found: ID does not exist" Oct 31 21:47:04.409543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.409469 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:47:04.409543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.409496 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm6lj\" (UniqueName: \"kubernetes.io/projected/8427cb0a-6554-4dfb-8779-647eac588f68-kube-api-access-wm6lj\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:47:04.409543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.409507 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8427cb0a-6554-4dfb-8779-647eac588f68-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:47:04.409543 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.409516 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8427cb0a-6554-4dfb-8779-647eac588f68-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:47:04.695480 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.694396 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq"] Oct 31 21:47:04.700494 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:04.700456 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-1f523a-68fb68d594-d2fnq"] Oct 31 21:47:05.475237 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:05.475196 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8427cb0a-6554-4dfb-8779-647eac588f68" path="/var/lib/kubelet/pods/8427cb0a-6554-4dfb-8779-647eac588f68/volumes" Oct 31 21:47:06.992686 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:06.992642 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn"] Oct 31 21:47:06.993339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:06.993301 5188 cpu_manager.go:414] "containerMap: removing container" podUID="8427cb0a-6554-4dfb-8779-647eac588f68" containerName="maas-api" Oct 31 21:47:06.993339 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:06.993331 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8427cb0a-6554-4dfb-8779-647eac588f68" containerName="maas-api" Oct 31 21:47:06.993483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:06.993347 5188 cpu_manager.go:414] "containerMap: removing container" podUID="1c2e64cf-ece3-4520-9f7e-0e39d51919f5" containerName="maas-api" Oct 31 21:47:06.993483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:06.993353 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2e64cf-ece3-4520-9f7e-0e39d51919f5" containerName="maas-api" Oct 31 21:47:06.993483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:06.993470 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="8427cb0a-6554-4dfb-8779-647eac588f68" containerName="maas-api" Oct 31 21:47:06.993483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:06.993481 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2e64cf-ece3-4520-9f7e-0e39d51919f5" containerName="maas-api" Oct 31 21:47:06.998613 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:06.998585 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.045961 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.045913 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn"] Oct 31 21:47:07.147298 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.147252 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.147298 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.147296 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.147556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.147346 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.147556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.147471 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.147556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.147542 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82q79\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-kube-api-access-82q79\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.147670 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.147582 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.147670 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.147659 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.147744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.147710 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.147776 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.147754 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.248813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.248722 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.248813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.248790 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.248813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.248813 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.248837 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249016 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249108 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82q79\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-kube-api-access-82q79\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249167 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249254 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249274 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249364 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249427 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249629 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249696 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249662 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.249696 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.249672 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.251585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.251551 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.251827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.251802 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.272484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.272444 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.272669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.272639 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82q79\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-kube-api-access-82q79\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.329101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.329062 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:07.663504 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:47:07.663471 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff44385a_0c7b_48c2_b43d_a83e007ad3ae.slice/crio-64d4b2c315d99ebb82f4b8a5b8d9f19e7036a0419c97fbcdd960e601cbdcb483 WatchSource:0}: Error finding container 64d4b2c315d99ebb82f4b8a5b8d9f19e7036a0419c97fbcdd960e601cbdcb483: Status 404 returned error can't find the container with id 64d4b2c315d99ebb82f4b8a5b8d9f19e7036a0419c97fbcdd960e601cbdcb483 Oct 31 21:47:07.664170 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.664105 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn"] Oct 31 21:47:07.666265 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.666231 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:47:07.666388 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.666298 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:47:07.666388 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:07.666344 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:47:08.361664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:08.361620 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" event={"ID":"ff44385a-0c7b-48c2-b43d-a83e007ad3ae","Type":"ContainerStarted","Data":"ed8c56ab5935075e85067f22a19f21a47911e7e3a18de4172d17f2ed79241c48"} Oct 31 21:47:08.362094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:08.361694 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" event={"ID":"ff44385a-0c7b-48c2-b43d-a83e007ad3ae","Type":"ContainerStarted","Data":"64d4b2c315d99ebb82f4b8a5b8d9f19e7036a0419c97fbcdd960e601cbdcb483"} Oct 31 21:47:08.408256 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:08.408178 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" podStartSLOduration=2.408154856 podStartE2EDuration="2.408154856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:08.403166131 +0000 UTC m=+1875.096123525" watchObservedRunningTime="2026-10-31 21:47:08.408154856 +0000 UTC m=+1875.101112250" Oct 31 21:47:09.330292 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:09.330247 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:09.332349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:09.332306 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 21:47:09.332502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:09.332369 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 21:47:10.329724 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:10.329672 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 21:47:10.330192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:10.329760 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 21:47:11.329936 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:11.329892 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 21:47:11.330405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:11.329954 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 21:47:12.030377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:12.030317 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn"] Oct 31 21:47:12.041808 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:12.041764 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7b302ad-qqtdn"] Oct 31 21:47:12.333874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:12.333769 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 21:47:12.334577 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:12.334533 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 21:47:13.330105 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:13.330063 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 21:47:13.330363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:13.330144 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 21:47:13.523848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:13.523806 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e626eaef-b437-4c91-8726-7de55755ca50" path="/var/lib/kubelet/pods/e626eaef-b437-4c91-8726-7de55755ca50/volumes" Oct 31 21:47:14.330266 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:14.330222 5188 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 21:47:14.330490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:14.330293 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 21:47:15.400366 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:15.400282 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:15.401069 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:15.401040 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:15.402148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:15.402119 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 21:47:19.301348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.293431 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl"] Oct 31 21:47:19.301348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.301148 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:19.334405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.331563 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl"] Oct 31 21:47:19.361350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.356857 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ad121b-dockercfg-px9k5\"" Oct 31 21:47:19.361350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.356948 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ad121b\"" Oct 31 21:47:19.513653 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.513609 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgtr4\" (UniqueName: \"kubernetes.io/projected/fa6222fe-f686-435c-bb08-c76a8d5de879-kube-api-access-qgtr4\") pod \"payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl\" (UID: \"fa6222fe-f686-435c-bb08-c76a8d5de879\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:19.513888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.513833 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6222fe-f686-435c-bb08-c76a8d5de879-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl\" (UID: \"fa6222fe-f686-435c-bb08-c76a8d5de879\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:19.616124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.614865 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6222fe-f686-435c-bb08-c76a8d5de879-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl\" (UID: \"fa6222fe-f686-435c-bb08-c76a8d5de879\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:19.616124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.615004 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgtr4\" (UniqueName: \"kubernetes.io/projected/fa6222fe-f686-435c-bb08-c76a8d5de879-kube-api-access-qgtr4\") pod \"payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl\" (UID: \"fa6222fe-f686-435c-bb08-c76a8d5de879\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:19.616124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.616040 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6222fe-f686-435c-bb08-c76a8d5de879-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl\" (UID: \"fa6222fe-f686-435c-bb08-c76a8d5de879\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:19.636641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.636604 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgtr4\" (UniqueName: \"kubernetes.io/projected/fa6222fe-f686-435c-bb08-c76a8d5de879-kube-api-access-qgtr4\") pod \"payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl\" (UID: \"fa6222fe-f686-435c-bb08-c76a8d5de879\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:19.864024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.863984 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn"] Oct 31 21:47:19.868614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.868532 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:19.889733 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.889695 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn"] Oct 31 21:47:19.924109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.921654 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e338f6f-7vfwn\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:19.924109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.921728 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfg9p\" (UniqueName: \"kubernetes.io/projected/73b5e1fe-ced7-426c-b422-b36d27089aa6-kube-api-access-kfg9p\") pod \"maas-api-revoke-sub-8e338f6f-7vfwn\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:19.924109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.921843 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e338f6f-7vfwn\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:19.924109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.923784 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:47:19.924109 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.923949 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:47:19.926622 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:19.926587 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:20.028846 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.028804 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e338f6f-7vfwn\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:20.029022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.028865 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfg9p\" (UniqueName: \"kubernetes.io/projected/73b5e1fe-ced7-426c-b422-b36d27089aa6-kube-api-access-kfg9p\") pod \"maas-api-revoke-sub-8e338f6f-7vfwn\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:20.029022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.028968 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e338f6f-7vfwn\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:20.030241 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.030210 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e338f6f-7vfwn\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:20.032844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.032811 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e338f6f-7vfwn\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:20.052358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.052314 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfg9p\" (UniqueName: \"kubernetes.io/projected/73b5e1fe-ced7-426c-b422-b36d27089aa6-kube-api-access-kfg9p\") pod \"maas-api-revoke-sub-8e338f6f-7vfwn\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:20.186375 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.186263 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:20.262691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.262651 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl"] Oct 31 21:47:20.502742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.502586 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" event={"ID":"fa6222fe-f686-435c-bb08-c76a8d5de879","Type":"ContainerStarted","Data":"33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158"} Oct 31 21:47:20.502742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.502656 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" event={"ID":"fa6222fe-f686-435c-bb08-c76a8d5de879","Type":"ContainerStarted","Data":"57bb1723c9b4ee90a8e275c1cfdc22702c7aac5abc30534f026cbded6115dfe6"} Oct 31 21:47:20.503215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.502901 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:20.503215 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.502951 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn"] Oct 31 21:47:20.504574 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:47:20.504532 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b5e1fe_ced7_426c_b422_b36d27089aa6.slice/crio-37011ec75acae7ea1170b7c73f4a6c4cbbffffdc546943b20a05a6ff7fc1b9da WatchSource:0}: Error finding container 37011ec75acae7ea1170b7c73f4a6c4cbbffffdc546943b20a05a6ff7fc1b9da: Status 404 returned error can't find the container with id 37011ec75acae7ea1170b7c73f4a6c4cbbffffdc546943b20a05a6ff7fc1b9da Oct 31 21:47:20.539456 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:20.539391 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" podStartSLOduration=1.539372677 podStartE2EDuration="1.539372677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:20.538371086 +0000 UTC m=+1887.231328479" watchObservedRunningTime="2026-10-31 21:47:20.539372677 +0000 UTC m=+1887.232330067" Oct 31 21:47:21.513258 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:21.513223 5188 generic.go:356] "Generic (PLEG): container finished" podID="73b5e1fe-ced7-426c-b422-b36d27089aa6" containerID="3683845c0a943ff2916ba990b3189ab4a6bd4f9eaea5013133aa4fba2474b674" exitCode=0 Oct 31 21:47:21.513700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:21.513305 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" event={"ID":"73b5e1fe-ced7-426c-b422-b36d27089aa6","Type":"ContainerDied","Data":"3683845c0a943ff2916ba990b3189ab4a6bd4f9eaea5013133aa4fba2474b674"} Oct 31 21:47:21.513700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:21.513411 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" event={"ID":"73b5e1fe-ced7-426c-b422-b36d27089aa6","Type":"ContainerStarted","Data":"37011ec75acae7ea1170b7c73f4a6c4cbbffffdc546943b20a05a6ff7fc1b9da"} Oct 31 21:47:22.725559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.725531 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:22.778245 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.778149 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-xks-ca\") pod \"73b5e1fe-ced7-426c-b422-b36d27089aa6\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " Oct 31 21:47:22.778444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.778244 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-openshift-ca\") pod \"73b5e1fe-ced7-426c-b422-b36d27089aa6\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " Oct 31 21:47:22.778444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.778402 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b5e1fe-ced7-426c-b422-b36d27089aa6-kube-api-access-kfg9p\" (UniqueName: \"kubernetes.io/projected/73b5e1fe-ced7-426c-b422-b36d27089aa6-kube-api-access-kfg9p\") pod \"73b5e1fe-ced7-426c-b422-b36d27089aa6\" (UID: \"73b5e1fe-ced7-426c-b422-b36d27089aa6\") " Oct 31 21:47:22.779937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.779905 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-openshift-ca" pod "73b5e1fe-ced7-426c-b422-b36d27089aa6" (UID: "73b5e1fe-ced7-426c-b422-b36d27089aa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:22.781784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.781754 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b5e1fe-ced7-426c-b422-b36d27089aa6-kube-api-access-kfg9p" pod "73b5e1fe-ced7-426c-b422-b36d27089aa6" (UID: "73b5e1fe-ced7-426c-b422-b36d27089aa6"). InnerVolumeSpecName "kube-api-access-kfg9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:22.782108 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.782080 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-xks-ca" pod "73b5e1fe-ced7-426c-b422-b36d27089aa6" (UID: "73b5e1fe-ced7-426c-b422-b36d27089aa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:22.880942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.880886 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:47:22.880942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.880946 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73b5e1fe-ced7-426c-b422-b36d27089aa6-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:47:22.881231 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:22.880962 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfg9p\" (UniqueName: \"kubernetes.io/projected/73b5e1fe-ced7-426c-b422-b36d27089aa6-kube-api-access-kfg9p\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:47:23.531368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:23.531333 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" Oct 31 21:47:23.531529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:23.531368 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn" event={"ID":"73b5e1fe-ced7-426c-b422-b36d27089aa6","Type":"ContainerDied","Data":"37011ec75acae7ea1170b7c73f4a6c4cbbffffdc546943b20a05a6ff7fc1b9da"} Oct 31 21:47:23.531529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:23.531417 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37011ec75acae7ea1170b7c73f4a6c4cbbffffdc546943b20a05a6ff7fc1b9da" Oct 31 21:47:31.517717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:31.517675 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:47:32.539123 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:47:32.539084 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:47:34.885710 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.885672 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n"] Oct 31 21:47:34.887037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.887007 5188 cpu_manager.go:414] "containerMap: removing container" podUID="73b5e1fe-ced7-426c-b422-b36d27089aa6" containerName="revoke-keys" Oct 31 21:47:34.887037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.887033 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b5e1fe-ced7-426c-b422-b36d27089aa6" containerName="revoke-keys" Oct 31 21:47:34.887275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.887253 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b5e1fe-ced7-426c-b422-b36d27089aa6" containerName="revoke-keys" Oct 31 21:47:34.891116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.891094 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:34.904223 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.904193 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ad121b-serving-cert\"" Oct 31 21:47:34.921715 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.921676 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n"] Oct 31 21:47:34.970364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.969219 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-metrics-certs\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:34.970364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.969279 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfd5de2-9903-47a4-b181-cd449243a179-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:34.970364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.969354 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbsn\" (UniqueName: \"kubernetes.io/projected/6cfd5de2-9903-47a4-b181-cd449243a179-kube-api-access-jdbsn\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:34.970364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:34.969452 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.086398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.082644 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.086398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.082817 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-metrics-certs\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.086398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.082855 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfd5de2-9903-47a4-b181-cd449243a179-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.086398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.082912 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbsn\" (UniqueName: \"kubernetes.io/projected/6cfd5de2-9903-47a4-b181-cd449243a179-kube-api-access-jdbsn\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.086398 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:47:35.082936 5188 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-serving-cert: secret "maas-api-e2e-praxis-b-ad121b-serving-cert" not found Oct 31 21:47:35.086398 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:47:35.083119 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls podName:6cfd5de2-9903-47a4-b181-cd449243a179 nodeName:}" failed. No retries permitted until 2026-10-31 21:47:35.583056362 +0000 UTC m=+1902.276013766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls") pod "maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" (UID: "6cfd5de2-9903-47a4-b181-cd449243a179") : secret "maas-api-e2e-praxis-b-ad121b-serving-cert" not found Oct 31 21:47:35.105345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.087872 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfd5de2-9903-47a4-b181-cd449243a179-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.111417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.109834 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-metrics-certs\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.112515 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.112471 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbsn\" (UniqueName: \"kubernetes.io/projected/6cfd5de2-9903-47a4-b181-cd449243a179-kube-api-access-jdbsn\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.589766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.589718 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.592229 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.592198 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:35.807147 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:35.807111 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:36.155067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:36.155027 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n"] Oct 31 21:47:36.155633 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:47:36.155405 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfd5de2_9903_47a4_b181_cd449243a179.slice/crio-bb474bf1a7c60a699cf7bab772b827f13c9590543266a1bf19958216284ef9b8 WatchSource:0}: Error finding container bb474bf1a7c60a699cf7bab772b827f13c9590543266a1bf19958216284ef9b8: Status 404 returned error can't find the container with id bb474bf1a7c60a699cf7bab772b827f13c9590543266a1bf19958216284ef9b8 Oct 31 21:47:36.641856 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:36.641819 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" event={"ID":"6cfd5de2-9903-47a4-b181-cd449243a179","Type":"ContainerStarted","Data":"50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab"} Oct 31 21:47:36.642065 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:36.641875 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" event={"ID":"6cfd5de2-9903-47a4-b181-cd449243a179","Type":"ContainerStarted","Data":"bb474bf1a7c60a699cf7bab772b827f13c9590543266a1bf19958216284ef9b8"} Oct 31 21:47:36.642065 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:36.641931 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:36.681365 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:36.681280 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" podStartSLOduration=2.3518371399999998 podStartE2EDuration="2.681264997s" totalImagesPullingTime="329.427857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:36.679607779 +0000 UTC m=+1903.372565173" watchObservedRunningTime="2026-10-31 21:47:36.681264997 +0000 UTC m=+1903.374222389" Oct 31 21:47:42.657362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:42.654751 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:47:50.162752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.162709 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh"] Oct 31 21:47:50.177025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.176987 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh"] Oct 31 21:47:50.177308 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.177106 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:47:50.191091 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.190939 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-b36e62\"" Oct 31 21:47:50.200382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.200310 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-b36e62-dockercfg-vw8g4\"" Oct 31 21:47:50.241858 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.241819 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8g7s\" (UniqueName: \"kubernetes.io/projected/64815867-ef42-4773-99ab-b5669ffe1265-kube-api-access-k8g7s\") pod \"payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh\" (UID: \"64815867-ef42-4773-99ab-b5669ffe1265\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:47:50.242085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.241883 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64815867-ef42-4773-99ab-b5669ffe1265-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh\" (UID: \"64815867-ef42-4773-99ab-b5669ffe1265\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:47:50.342968 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.342920 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8g7s\" (UniqueName: \"kubernetes.io/projected/64815867-ef42-4773-99ab-b5669ffe1265-kube-api-access-k8g7s\") pod \"payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh\" (UID: \"64815867-ef42-4773-99ab-b5669ffe1265\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:47:50.343185 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.342999 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64815867-ef42-4773-99ab-b5669ffe1265-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh\" (UID: \"64815867-ef42-4773-99ab-b5669ffe1265\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:47:50.344086 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.344053 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64815867-ef42-4773-99ab-b5669ffe1265-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh\" (UID: \"64815867-ef42-4773-99ab-b5669ffe1265\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:47:50.365182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.365147 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8g7s\" (UniqueName: \"kubernetes.io/projected/64815867-ef42-4773-99ab-b5669ffe1265-kube-api-access-k8g7s\") pod \"payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh\" (UID: \"64815867-ef42-4773-99ab-b5669ffe1265\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:47:50.501000 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.500908 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:47:50.810150 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:47:50.810105 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64815867_ef42_4773_99ab_b5669ffe1265.slice/crio-455ffca2e2307013e0c18148d4e3d592b7cfc102ad1c4a52a35c42e1975dc188 WatchSource:0}: Error finding container 455ffca2e2307013e0c18148d4e3d592b7cfc102ad1c4a52a35c42e1975dc188: Status 404 returned error can't find the container with id 455ffca2e2307013e0c18148d4e3d592b7cfc102ad1c4a52a35c42e1975dc188 Oct 31 21:47:50.810582 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.810553 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh"] Oct 31 21:47:50.850874 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:50.850840 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" event={"ID":"64815867-ef42-4773-99ab-b5669ffe1265","Type":"ContainerStarted","Data":"455ffca2e2307013e0c18148d4e3d592b7cfc102ad1c4a52a35c42e1975dc188"} Oct 31 21:47:51.869708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:51.869672 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" event={"ID":"64815867-ef42-4773-99ab-b5669ffe1265","Type":"ContainerStarted","Data":"84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28"} Oct 31 21:47:51.870214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:51.869789 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:47:51.905896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:51.905831 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" podStartSLOduration=1.9058106750000001 podStartE2EDuration="1.905810675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:51.905078122 +0000 UTC m=+1918.598035516" watchObservedRunningTime="2026-10-31 21:47:51.905810675 +0000 UTC m=+1918.598768069" Oct 31 21:47:56.437076 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:56.437041 5188 scope.go:122] "RemoveContainer" containerID="4cbedf6a214bb6c4dc9af901b097b6468372fc5fb92cfc321e47918cbed24712" Oct 31 21:47:56.449283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:56.449255 5188 scope.go:122] "RemoveContainer" containerID="720f8bf66e7823ae710309120d806994bcb46eb1783106a7732b40f351e05b46" Oct 31 21:47:58.541355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:47:58.540198 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp"] Oct 31 21:48:00.950124 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:00.950078 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf"] Oct 31 21:48:00.950781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:00.950386 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" podUID="752aebb7-9f3c-4c16-aea5-e08ff2a9d312" containerName="maas-api" containerID="cri-o://b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8" gracePeriod=30 Oct 31 21:48:01.289940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.289907 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:48:01.389347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.380878 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-metrics-certs\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-metrics-certs\") pod \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " Oct 31 21:48:01.389347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.380926 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-openshift-service-ca\") pod \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " Oct 31 21:48:01.389347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.380997 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-kube-api-access-bkk8v\" (UniqueName: \"kubernetes.io/projected/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-kube-api-access-bkk8v\") pod \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " Oct 31 21:48:01.389347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.381137 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-maas-api-tls\") pod \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\" (UID: \"752aebb7-9f3c-4c16-aea5-e08ff2a9d312\") " Oct 31 21:48:01.389347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.386097 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-maas-api-tls" pod "752aebb7-9f3c-4c16-aea5-e08ff2a9d312" (UID: "752aebb7-9f3c-4c16-aea5-e08ff2a9d312"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:48:01.412403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.411230 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-openshift-service-ca" pod "752aebb7-9f3c-4c16-aea5-e08ff2a9d312" (UID: "752aebb7-9f3c-4c16-aea5-e08ff2a9d312"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:01.413811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.413734 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-kube-api-access-bkk8v" pod "752aebb7-9f3c-4c16-aea5-e08ff2a9d312" (UID: "752aebb7-9f3c-4c16-aea5-e08ff2a9d312"). InnerVolumeSpecName "kube-api-access-bkk8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:01.419917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.419848 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-metrics-certs" pod "752aebb7-9f3c-4c16-aea5-e08ff2a9d312" (UID: "752aebb7-9f3c-4c16-aea5-e08ff2a9d312"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:48:01.482744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.482710 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:01.482744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.482741 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:01.482744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.482753 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:01.482744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:01.482762 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkk8v\" (UniqueName: \"kubernetes.io/projected/752aebb7-9f3c-4c16-aea5-e08ff2a9d312-kube-api-access-bkk8v\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:02.003213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.003177 5188 generic.go:356] "Generic (PLEG): container finished" podID="752aebb7-9f3c-4c16-aea5-e08ff2a9d312" containerID="b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8" exitCode=0 Oct 31 21:48:02.003728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.003231 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" Oct 31 21:48:02.003728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.003269 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" event={"ID":"752aebb7-9f3c-4c16-aea5-e08ff2a9d312","Type":"ContainerDied","Data":"b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8"} Oct 31 21:48:02.003728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.003379 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf" event={"ID":"752aebb7-9f3c-4c16-aea5-e08ff2a9d312","Type":"ContainerDied","Data":"02b7147231f694baa6ff018d609ed1e1ec34a51a788fdf15aba6faf2e2a2e2f0"} Oct 31 21:48:02.003728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.003412 5188 scope.go:122] "RemoveContainer" containerID="b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8" Oct 31 21:48:02.015148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.015124 5188 scope.go:122] "RemoveContainer" containerID="b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8" Oct 31 21:48:02.015564 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:48:02.015541 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8\": container with ID starting with b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8 not found: ID does not exist" containerID="b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8" Oct 31 21:48:02.015657 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.015583 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8"} err="failed to get container status \"b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8\": rpc error: code = NotFound desc = could not find container \"b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8\": container with ID starting with b719d81c027d695aa24ac6f974aa6634571f0d2bb0f2626b7d5e058d8c9dd8a8 not found: ID does not exist" Oct 31 21:48:02.049801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.049763 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf"] Oct 31 21:48:02.059309 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.059269 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-9c8b33-67c5b44f98-hz7jf"] Oct 31 21:48:02.542192 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:48:02.542152 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:48:02.883063 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:02.883031 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 21:48:03.476070 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:03.476034 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752aebb7-9f3c-4c16-aea5-e08ff2a9d312" path="/var/lib/kubelet/pods/752aebb7-9f3c-4c16-aea5-e08ff2a9d312/volumes" Oct 31 21:48:03.541557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:03.541504 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" podUID="5460964d-80dd-41a2-89f1-3d5915432dd0" containerName="payload-processing" containerID="cri-o://130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69" gracePeriod=25 Oct 31 21:48:03.878013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:03.877976 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:48:04.023094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.023006 5188 generic.go:356] "Generic (PLEG): container finished" podID="5460964d-80dd-41a2-89f1-3d5915432dd0" containerID="130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69" exitCode=0 Oct 31 21:48:04.023094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.023082 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" Oct 31 21:48:04.023404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.023096 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" event={"ID":"5460964d-80dd-41a2-89f1-3d5915432dd0","Type":"ContainerDied","Data":"130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69"} Oct 31 21:48:04.023404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.023171 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp" event={"ID":"5460964d-80dd-41a2-89f1-3d5915432dd0","Type":"ContainerDied","Data":"27e7da4ba87e8b9cce5d82e012e59d78296b712b64009084f74eee58394e3dcb"} Oct 31 21:48:04.023404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.023204 5188 scope.go:122] "RemoveContainer" containerID="130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69" Oct 31 21:48:04.028934 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.028898 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5460964d-80dd-41a2-89f1-3d5915432dd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460964d-80dd-41a2-89f1-3d5915432dd0-plugins-config-volume\") pod \"5460964d-80dd-41a2-89f1-3d5915432dd0\" (UID: \"5460964d-80dd-41a2-89f1-3d5915432dd0\") " Oct 31 21:48:04.029092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.028941 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5460964d-80dd-41a2-89f1-3d5915432dd0-kube-api-access-qhb2v\" (UniqueName: \"kubernetes.io/projected/5460964d-80dd-41a2-89f1-3d5915432dd0-kube-api-access-qhb2v\") pod \"5460964d-80dd-41a2-89f1-3d5915432dd0\" (UID: \"5460964d-80dd-41a2-89f1-3d5915432dd0\") " Oct 31 21:48:04.029397 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.029366 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5460964d-80dd-41a2-89f1-3d5915432dd0-plugins-config-volume" pod "5460964d-80dd-41a2-89f1-3d5915432dd0" (UID: "5460964d-80dd-41a2-89f1-3d5915432dd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:04.029674 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.029652 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5460964d-80dd-41a2-89f1-3d5915432dd0-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:04.031440 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.031408 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5460964d-80dd-41a2-89f1-3d5915432dd0-kube-api-access-qhb2v" pod "5460964d-80dd-41a2-89f1-3d5915432dd0" (UID: "5460964d-80dd-41a2-89f1-3d5915432dd0"). InnerVolumeSpecName "kube-api-access-qhb2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:04.034708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.034675 5188 scope.go:122] "RemoveContainer" containerID="130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69" Oct 31 21:48:04.035104 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:48:04.035079 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69\": container with ID starting with 130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69 not found: ID does not exist" containerID="130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69" Oct 31 21:48:04.035200 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.035110 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69"} err="failed to get container status \"130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69\": rpc error: code = NotFound desc = could not find container \"130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69\": container with ID starting with 130f471a824b8b6c5f5a20e89e9961e3448152eb211c328e23d193eff00dfc69 not found: ID does not exist" Oct 31 21:48:04.131105 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.131061 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhb2v\" (UniqueName: \"kubernetes.io/projected/5460964d-80dd-41a2-89f1-3d5915432dd0-kube-api-access-qhb2v\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:04.370626 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.370587 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp"] Oct 31 21:48:04.379726 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:04.379691 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9c8b33-59cdb566d5-n8frp"] Oct 31 21:48:05.475046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:05.475003 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5460964d-80dd-41a2-89f1-3d5915432dd0" path="/var/lib/kubelet/pods/5460964d-80dd-41a2-89f1-3d5915432dd0/volumes" Oct 31 21:48:07.076501 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.076455 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l"] Oct 31 21:48:07.077283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.077258 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5460964d-80dd-41a2-89f1-3d5915432dd0" containerName="payload-processing" Oct 31 21:48:07.077283 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.077283 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5460964d-80dd-41a2-89f1-3d5915432dd0" containerName="payload-processing" Oct 31 21:48:07.077495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.077304 5188 cpu_manager.go:414] "containerMap: removing container" podUID="752aebb7-9f3c-4c16-aea5-e08ff2a9d312" containerName="maas-api" Oct 31 21:48:07.077495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.077309 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752aebb7-9f3c-4c16-aea5-e08ff2a9d312" containerName="maas-api" Oct 31 21:48:07.077593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.077567 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5460964d-80dd-41a2-89f1-3d5915432dd0" containerName="payload-processing" Oct 31 21:48:07.077593 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.077589 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="752aebb7-9f3c-4c16-aea5-e08ff2a9d312" containerName="maas-api" Oct 31 21:48:07.083468 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.083440 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.107967 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.107163 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-b36e62-serving-cert\"" Oct 31 21:48:07.113580 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.113541 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l"] Oct 31 21:48:07.270828 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.270783 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.271054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.270934 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab743db-44f2-455e-ba49-4d0afea44c71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.271054 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.271023 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpk5\" (UniqueName: \"kubernetes.io/projected/1ab743db-44f2-455e-ba49-4d0afea44c71-kube-api-access-hwpk5\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.271168 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.271110 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.372799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.372748 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpk5\" (UniqueName: \"kubernetes.io/projected/1ab743db-44f2-455e-ba49-4d0afea44c71-kube-api-access-hwpk5\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.373046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.372844 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.373046 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.373041 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.373866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.373121 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab743db-44f2-455e-ba49-4d0afea44c71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.374045 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.374006 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab743db-44f2-455e-ba49-4d0afea44c71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.409189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.406459 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpk5\" (UniqueName: \"kubernetes.io/projected/1ab743db-44f2-455e-ba49-4d0afea44c71-kube-api-access-hwpk5\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.410198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.410165 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.410492 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.410473 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:07.701023 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:07.700930 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:08.026690 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:08.025305 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l"] Oct 31 21:48:08.064644 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:08.064597 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" event={"ID":"1ab743db-44f2-455e-ba49-4d0afea44c71","Type":"ContainerStarted","Data":"14c0b988ea95aa56d3b4517ba12e3f6cb183c618280b4dfffebec40c0f7666a9"} Oct 31 21:48:09.042938 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.040434 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5"] Oct 31 21:48:09.086735 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.086680 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" containerID="cri-o://2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e" gracePeriod=30 Oct 31 21:48:09.089036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.088996 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" event={"ID":"1ab743db-44f2-455e-ba49-4d0afea44c71","Type":"ContainerStarted","Data":"7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b"} Oct 31 21:48:09.089247 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.089230 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:09.168198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.167646 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" podStartSLOduration=1.728995369 podStartE2EDuration="2.167623611s" totalImagesPullingTime="438.628242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:09.164729593 +0000 UTC m=+1935.857686999" watchObservedRunningTime="2026-10-31 21:48:09.167623611 +0000 UTC m=+1935.860581005" Oct 31 21:48:09.753655 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.753616 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55"] Oct 31 21:48:09.758470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.758440 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.790759 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.786139 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55"] Oct 31 21:48:09.841637 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.841595 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.841857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.841641 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjw9\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-kube-api-access-lgjw9\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.841857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.841749 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.841857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.841815 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.842016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.841911 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef870d9-ae63-4135-8a64-b4adab4ec07c-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.842016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.841981 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.842094 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.842022 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.842131 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.842093 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.842166 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.842146 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.943945 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.943906 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef870d9-ae63-4135-8a64-b4adab4ec07c-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.943964 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944035 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944091 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944126 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944214 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944239 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjw9\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-kube-api-access-lgjw9\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944287 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944351 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944555 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944665 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944742 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944871 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944857 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.944935 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.944918 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef870d9-ae63-4135-8a64-b4adab4ec07c-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.946985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.946939 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.947364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.947302 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.966155 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.966120 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:09.966362 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:09.966338 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjw9\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-kube-api-access-lgjw9\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:10.082359 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:10.082223 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:10.415233 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:10.415195 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55"] Oct 31 21:48:10.415936 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:48:10.415904 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef870d9_ae63_4135_8a64_b4adab4ec07c.slice/crio-00eaa6c5cab6dfe8528866f7e7ff92e0c30a130ad5e2f7d0b10cce923e4bb069 WatchSource:0}: Error finding container 00eaa6c5cab6dfe8528866f7e7ff92e0c30a130ad5e2f7d0b10cce923e4bb069: Status 404 returned error can't find the container with id 00eaa6c5cab6dfe8528866f7e7ff92e0c30a130ad5e2f7d0b10cce923e4bb069 Oct 31 21:48:10.419305 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:10.419268 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:48:10.419465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:10.419379 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:48:10.419465 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:10.419428 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:48:11.140471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:11.140403 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" event={"ID":"fef870d9-ae63-4135-8a64-b4adab4ec07c","Type":"ContainerStarted","Data":"159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff"} Oct 31 21:48:11.141022 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:11.140557 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" event={"ID":"fef870d9-ae63-4135-8a64-b4adab4ec07c","Type":"ContainerStarted","Data":"00eaa6c5cab6dfe8528866f7e7ff92e0c30a130ad5e2f7d0b10cce923e4bb069"} Oct 31 21:48:11.205345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:11.205266 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" podStartSLOduration=2.205246995 podStartE2EDuration="2.205246995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:11.192169637 +0000 UTC m=+1937.885127031" watchObservedRunningTime="2026-10-31 21:48:11.205246995 +0000 UTC m=+1937.898204387" Oct 31 21:48:12.086900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.086861 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:12.087316 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.087289 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 21:48:12.087614 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.087570 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 21:48:12.411643 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.411595 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t"] Oct 31 21:48:12.435143 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.434427 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t"] Oct 31 21:48:12.435143 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.434585 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.450407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.448580 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-0877f5-serving-cert\"" Oct 31 21:48:12.523643 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.523596 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-kube-api-access-w4zcq\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.523872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.523797 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.523872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.523848 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.523998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.523911 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.625649 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.625604 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-kube-api-access-w4zcq\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.625827 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.625711 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.625877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.625855 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.625940 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.625924 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.626516 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.626483 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.628722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.628689 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.628975 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.628956 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.647424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.647392 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-kube-api-access-w4zcq\") pod \"maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:12.760648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:12.760559 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:13.104565 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:13.096794 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 21:48:13.104565 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:13.096907 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 21:48:13.210120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:13.210078 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t"] Oct 31 21:48:13.961723 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:13.961674 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nq6b5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:48:13.962396 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:13.961744 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:48:14.097579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.091890 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 21:48:14.097579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.091958 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 21:48:14.169445 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.169395 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" event={"ID":"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7","Type":"ContainerStarted","Data":"b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26"} Oct 31 21:48:14.169665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.169473 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" event={"ID":"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7","Type":"ContainerStarted","Data":"c4d2763a35f0498155e466655db34e6c7671ec483567faa7e3ec0bc012107725"} Oct 31 21:48:14.171230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.170571 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:14.218917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.218849 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" podStartSLOduration=1.900726991 podStartE2EDuration="2.218828408s" totalImagesPullingTime="318.101417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:14.21480886 +0000 UTC m=+1940.907766254" watchObservedRunningTime="2026-10-31 21:48:14.218828408 +0000 UTC m=+1940.911785797" Oct 31 21:48:14.541699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.540291 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:48:14.635671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.635421 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-kube-api-access-vnvvv\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-kube-api-access-vnvvv\") pod \"773ae86a-2248-47ba-a028-474ecbb56a97\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " Oct 31 21:48:14.635671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.635624 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-socket\") pod \"773ae86a-2248-47ba-a028-474ecbb56a97\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " Oct 31 21:48:14.635962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.635870 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-envoy\") pod \"773ae86a-2248-47ba-a028-474ecbb56a97\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " Oct 31 21:48:14.635962 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.635911 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/773ae86a-2248-47ba-a028-474ecbb56a97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773ae86a-2248-47ba-a028-474ecbb56a97-istio-podinfo\") pod \"773ae86a-2248-47ba-a028-474ecbb56a97\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " Oct 31 21:48:14.636072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.635990 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-istio-token\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-istio-token\") pod \"773ae86a-2248-47ba-a028-474ecbb56a97\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " Oct 31 21:48:14.636072 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.636031 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-certs\") pod \"773ae86a-2248-47ba-a028-474ecbb56a97\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " Oct 31 21:48:14.636359 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.636276 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773ae86a-2248-47ba-a028-474ecbb56a97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773ae86a-2248-47ba-a028-474ecbb56a97-istiod-ca-cert\") pod \"773ae86a-2248-47ba-a028-474ecbb56a97\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " Oct 31 21:48:14.639621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.636485 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-data\") pod \"773ae86a-2248-47ba-a028-474ecbb56a97\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " Oct 31 21:48:14.639621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.636524 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-credential-socket\") pod \"773ae86a-2248-47ba-a028-474ecbb56a97\" (UID: \"773ae86a-2248-47ba-a028-474ecbb56a97\") " Oct 31 21:48:14.639621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.638852 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-certs" pod "773ae86a-2248-47ba-a028-474ecbb56a97" (UID: "773ae86a-2248-47ba-a028-474ecbb56a97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:14.639621 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.639185 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773ae86a-2248-47ba-a028-474ecbb56a97-istiod-ca-cert" pod "773ae86a-2248-47ba-a028-474ecbb56a97" (UID: "773ae86a-2248-47ba-a028-474ecbb56a97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:14.639994 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.639640 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-credential-socket" pod "773ae86a-2248-47ba-a028-474ecbb56a97" (UID: "773ae86a-2248-47ba-a028-474ecbb56a97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:14.640057 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.639974 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-data" pod "773ae86a-2248-47ba-a028-474ecbb56a97" (UID: "773ae86a-2248-47ba-a028-474ecbb56a97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:14.640496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.640225 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-socket" pod "773ae86a-2248-47ba-a028-474ecbb56a97" (UID: "773ae86a-2248-47ba-a028-474ecbb56a97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:14.641966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.641802 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:14.641966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.641838 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:14.641966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.641854 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773ae86a-2248-47ba-a028-474ecbb56a97-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:14.641966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.641868 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:14.641966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.641882 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:14.808405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.805592 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/773ae86a-2248-47ba-a028-474ecbb56a97-istio-podinfo" pod "773ae86a-2248-47ba-a028-474ecbb56a97" (UID: "773ae86a-2248-47ba-a028-474ecbb56a97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:48:14.808405 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.805816 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-istio-token" pod "773ae86a-2248-47ba-a028-474ecbb56a97" (UID: "773ae86a-2248-47ba-a028-474ecbb56a97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:14.817760 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.816617 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-kube-api-access-vnvvv" pod "773ae86a-2248-47ba-a028-474ecbb56a97" (UID: "773ae86a-2248-47ba-a028-474ecbb56a97"). InnerVolumeSpecName "kube-api-access-vnvvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:14.832034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.831956 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-envoy" pod "773ae86a-2248-47ba-a028-474ecbb56a97" (UID: "773ae86a-2248-47ba-a028-474ecbb56a97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:14.877356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.873472 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773ae86a-2248-47ba-a028-474ecbb56a97-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:14.877356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.873522 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773ae86a-2248-47ba-a028-474ecbb56a97-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:14.877356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.873538 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:14.877356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:14.873554 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnvvv\" (UniqueName: \"kubernetes.io/projected/773ae86a-2248-47ba-a028-474ecbb56a97-kube-api-access-vnvvv\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:15.097779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.082891 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 21:48:15.097779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.083109 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 21:48:15.109064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.108856 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 21:48:15.209937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.208745 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr"] Oct 31 21:48:15.214351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.211446 5188 generic.go:356] "Generic (PLEG): container finished" podID="773ae86a-2248-47ba-a028-474ecbb56a97" containerID="2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e" exitCode=0 Oct 31 21:48:15.214351 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.212659 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" Oct 31 21:48:15.214721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.214412 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" event={"ID":"773ae86a-2248-47ba-a028-474ecbb56a97","Type":"ContainerDied","Data":"2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e"} Oct 31 21:48:15.214721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.214487 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5" event={"ID":"773ae86a-2248-47ba-a028-474ecbb56a97","Type":"ContainerDied","Data":"9abed1318be9693f951cf5fd1f96123eb9da6336a66a0748e422b42eff7fc509"} Oct 31 21:48:15.214721 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.214520 5188 scope.go:122] "RemoveContainer" containerID="2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e" Oct 31 21:48:15.237079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.235485 5188 scope.go:122] "RemoveContainer" containerID="2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e" Oct 31 21:48:15.237079 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:48:15.235858 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e\": container with ID starting with 2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e not found: ID does not exist" containerID="2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e" Oct 31 21:48:15.237079 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.235886 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e"} err="failed to get container status \"2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e\": rpc error: code = NotFound desc = could not find container \"2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e\": container with ID starting with 2de0bf3e9546224fb151a536ebfbdfd20e052cb56710db2b136e4cb4db64c69e not found: ID does not exist" Oct 31 21:48:15.239828 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.239795 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d"] Oct 31 21:48:15.349210 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.349110 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5"] Oct 31 21:48:15.355767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.355722 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nq6b5"] Oct 31 21:48:15.507642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:15.507603 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" path="/var/lib/kubelet/pods/773ae86a-2248-47ba-a028-474ecbb56a97/volumes" Oct 31 21:48:16.095359 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:16.095238 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 21:48:16.095359 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:16.095294 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 21:48:17.083209 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.083170 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 21:48:17.083781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.083246 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 21:48:17.233341 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.233284 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p"] Oct 31 21:48:17.234581 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.234547 5188 cpu_manager.go:414] "containerMap: removing container" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" Oct 31 21:48:17.234581 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.234579 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" Oct 31 21:48:17.234756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.234744 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="773ae86a-2248-47ba-a028-474ecbb56a97" containerName="istio-proxy" Oct 31 21:48:17.239093 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.239065 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.258425 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.258381 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p"] Oct 31 21:48:17.263313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.261984 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c744c0\"/\"openshift-service-ca.crt\"" Oct 31 21:48:17.263313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.262021 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c744c0\"/\"kube-root-ca.crt\"" Oct 31 21:48:17.272182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.272053 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c744c0\"/\"default-dockercfg-xcxlq\"" Oct 31 21:48:17.272182 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.272053 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c744c0\"/\"auth-test-model-c744c0-kserve-self-signed-certs\"" Oct 31 21:48:17.364250 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.364214 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-home\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.364467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.364274 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-tmp-dir\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.364467 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.364394 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-model-cache\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.364565 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.364471 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaebe34-779f-4ea3-b254-51511cb1c455-tls-certs\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.364611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.364586 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-dshm\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.364647 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.364632 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrq29\" (UniqueName: \"kubernetes.io/projected/bbaebe34-779f-4ea3-b254-51511cb1c455-kube-api-access-rrq29\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.470394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.469900 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-dshm\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.478583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.478537 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrq29\" (UniqueName: \"kubernetes.io/projected/bbaebe34-779f-4ea3-b254-51511cb1c455-kube-api-access-rrq29\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.478960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.478679 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-home\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.478960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.478739 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-tmp-dir\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.478960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.478819 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-model-cache\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.478960 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.478956 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaebe34-779f-4ea3-b254-51511cb1c455-tls-certs\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.490391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.480062 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-home\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.490391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.480741 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-model-cache\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.490391 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.482718 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-tmp-dir\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.521763 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.518055 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaebe34-779f-4ea3-b254-51511cb1c455-tls-certs\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.521763 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.518679 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-dshm\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.525601 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.525564 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrq29\" (UniqueName: \"kubernetes.io/projected/bbaebe34-779f-4ea3-b254-51511cb1c455-kube-api-access-rrq29\") pod \"auth-test-model-c744c0-kserve-ddf7887cc-72w7p\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.562345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.557420 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:17.929646 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:17.929609 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p"] Oct 31 21:48:17.930562 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:48:17.930531 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbaebe34_779f_4ea3_b254_51511cb1c455.slice/crio-f62ba0648dc80b33915fa9045fe3cd577a1c172918df341d04bbe187933fbdad WatchSource:0}: Error finding container f62ba0648dc80b33915fa9045fe3cd577a1c172918df341d04bbe187933fbdad: Status 404 returned error can't find the container with id f62ba0648dc80b33915fa9045fe3cd577a1c172918df341d04bbe187933fbdad Oct 31 21:48:18.244974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:18.244872 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" event={"ID":"bbaebe34-779f-4ea3-b254-51511cb1c455","Type":"ContainerStarted","Data":"16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940"} Oct 31 21:48:18.244974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:18.244933 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" event={"ID":"bbaebe34-779f-4ea3-b254-51511cb1c455","Type":"ContainerStarted","Data":"f62ba0648dc80b33915fa9045fe3cd577a1c172918df341d04bbe187933fbdad"} Oct 31 21:48:18.283957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:18.283822 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" podStartSLOduration=1.283801664 podStartE2EDuration="1.283801664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:18.281533442 +0000 UTC m=+1944.974490839" watchObservedRunningTime="2026-10-31 21:48:18.283801664 +0000 UTC m=+1944.976759057" Oct 31 21:48:18.548526 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:18.547819 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:18.548685 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:18.548572 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:18.549576 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:18.549545 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:48:18.557955 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:18.557913 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:20.209512 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.209441 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" podUID="9966a67d-a22f-451f-987a-448ca7b01aa9" containerName="payload-processing" containerID="cri-o://f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e" gracePeriod=25 Oct 31 21:48:20.240813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.240748 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" podUID="6800f9ca-3103-4c97-82ae-ff7aaa223df3" containerName="payload-pre-processing" containerID="cri-o://55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062" gracePeriod=25 Oct 31 21:48:20.556488 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.556453 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:48:20.587963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.587930 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:48:20.684055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.682478 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9966a67d-a22f-451f-987a-448ca7b01aa9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9966a67d-a22f-451f-987a-448ca7b01aa9-plugins-config-volume\") pod \"9966a67d-a22f-451f-987a-448ca7b01aa9\" (UID: \"9966a67d-a22f-451f-987a-448ca7b01aa9\") " Oct 31 21:48:20.684055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.682563 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6800f9ca-3103-4c97-82ae-ff7aaa223df3-kube-api-access-2886v\" (UniqueName: \"kubernetes.io/projected/6800f9ca-3103-4c97-82ae-ff7aaa223df3-kube-api-access-2886v\") pod \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\" (UID: \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\") " Oct 31 21:48:20.684055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.682643 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6800f9ca-3103-4c97-82ae-ff7aaa223df3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6800f9ca-3103-4c97-82ae-ff7aaa223df3-plugins-config-volume\") pod \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\" (UID: \"6800f9ca-3103-4c97-82ae-ff7aaa223df3\") " Oct 31 21:48:20.684055 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.682719 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9966a67d-a22f-451f-987a-448ca7b01aa9-kube-api-access-k25vd\" (UniqueName: \"kubernetes.io/projected/9966a67d-a22f-451f-987a-448ca7b01aa9-kube-api-access-k25vd\") pod \"9966a67d-a22f-451f-987a-448ca7b01aa9\" (UID: \"9966a67d-a22f-451f-987a-448ca7b01aa9\") " Oct 31 21:48:20.685847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.685805 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6800f9ca-3103-4c97-82ae-ff7aaa223df3-plugins-config-volume" pod "6800f9ca-3103-4c97-82ae-ff7aaa223df3" (UID: "6800f9ca-3103-4c97-82ae-ff7aaa223df3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:20.686269 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.686227 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9966a67d-a22f-451f-987a-448ca7b01aa9-plugins-config-volume" pod "9966a67d-a22f-451f-987a-448ca7b01aa9" (UID: "9966a67d-a22f-451f-987a-448ca7b01aa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:20.690042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.689991 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9966a67d-a22f-451f-987a-448ca7b01aa9-kube-api-access-k25vd" pod "9966a67d-a22f-451f-987a-448ca7b01aa9" (UID: "9966a67d-a22f-451f-987a-448ca7b01aa9"). InnerVolumeSpecName "kube-api-access-k25vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:20.690184 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.690117 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6800f9ca-3103-4c97-82ae-ff7aaa223df3-kube-api-access-2886v" pod "6800f9ca-3103-4c97-82ae-ff7aaa223df3" (UID: "6800f9ca-3103-4c97-82ae-ff7aaa223df3"). InnerVolumeSpecName "kube-api-access-2886v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:20.785301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.785207 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9966a67d-a22f-451f-987a-448ca7b01aa9-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:20.785301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.785244 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2886v\" (UniqueName: \"kubernetes.io/projected/6800f9ca-3103-4c97-82ae-ff7aaa223df3-kube-api-access-2886v\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:20.785301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.785255 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6800f9ca-3103-4c97-82ae-ff7aaa223df3-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:20.785301 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:20.785264 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25vd\" (UniqueName: \"kubernetes.io/projected/9966a67d-a22f-451f-987a-448ca7b01aa9-kube-api-access-k25vd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.229483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.229454 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 21:48:21.275705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.275662 5188 generic.go:356] "Generic (PLEG): container finished" podID="9966a67d-a22f-451f-987a-448ca7b01aa9" containerID="f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e" exitCode=0 Oct 31 21:48:21.275705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.275707 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" Oct 31 21:48:21.275957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.275742 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" event={"ID":"9966a67d-a22f-451f-987a-448ca7b01aa9","Type":"ContainerDied","Data":"f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e"} Oct 31 21:48:21.275957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.275805 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr" event={"ID":"9966a67d-a22f-451f-987a-448ca7b01aa9","Type":"ContainerDied","Data":"fb1321781979a4ab896482e185ba84d64aee6e8e9e66257f5f9ad8cf9e2e17f1"} Oct 31 21:48:21.275957 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.275839 5188 scope.go:122] "RemoveContainer" containerID="f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e" Oct 31 21:48:21.279916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.279881 5188 generic.go:356] "Generic (PLEG): container finished" podID="6800f9ca-3103-4c97-82ae-ff7aaa223df3" containerID="55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062" exitCode=0 Oct 31 21:48:21.280075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.280010 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" event={"ID":"6800f9ca-3103-4c97-82ae-ff7aaa223df3","Type":"ContainerDied","Data":"55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062"} Oct 31 21:48:21.280075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.280064 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" event={"ID":"6800f9ca-3103-4c97-82ae-ff7aaa223df3","Type":"ContainerDied","Data":"991559a2eb434ca3783a83a7eb4f72b334a05b98b40fe980683b7abad17380c8"} Oct 31 21:48:21.280210 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.280143 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d" Oct 31 21:48:21.296059 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.296035 5188 scope.go:122] "RemoveContainer" containerID="f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e" Oct 31 21:48:21.296698 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:48:21.296527 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e\": container with ID starting with f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e not found: ID does not exist" containerID="f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e" Oct 31 21:48:21.296698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.296574 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e"} err="failed to get container status \"f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e\": rpc error: code = NotFound desc = could not find container \"f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e\": container with ID starting with f2fd68e4fce5a7f3982933b27efbf6517c9fca6a7595d18113ae398e6bf8472e not found: ID does not exist" Oct 31 21:48:21.296698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.296603 5188 scope.go:122] "RemoveContainer" containerID="55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062" Oct 31 21:48:21.313942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.313914 5188 scope.go:122] "RemoveContainer" containerID="55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062" Oct 31 21:48:21.314514 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:48:21.314485 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062\": container with ID starting with 55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062 not found: ID does not exist" containerID="55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062" Oct 31 21:48:21.314630 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.314521 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062"} err="failed to get container status \"55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062\": rpc error: code = NotFound desc = could not find container \"55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062\": container with ID starting with 55b26dcb96f7d18c5de571bb2d3243be065570f7282cf551874ef37cbaa80062 not found: ID does not exist" Oct 31 21:48:21.338916 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.338882 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d"] Oct 31 21:48:21.343565 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.343527 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b01ff6-7d7ccdc8ff-cxl4d"] Oct 31 21:48:21.368981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.368941 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr"] Oct 31 21:48:21.376432 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.376395 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b01ff6-7688d65785-bgzkr"] Oct 31 21:48:21.475483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.475440 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6800f9ca-3103-4c97-82ae-ff7aaa223df3" path="/var/lib/kubelet/pods/6800f9ca-3103-4c97-82ae-ff7aaa223df3/volumes" Oct 31 21:48:21.476126 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:21.476100 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9966a67d-a22f-451f-987a-448ca7b01aa9" path="/var/lib/kubelet/pods/9966a67d-a22f-451f-987a-448ca7b01aa9/volumes" Oct 31 21:48:23.725494 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:23.725445 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd"] Oct 31 21:48:23.726040 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:23.725759 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" podUID="6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" containerName="maas-api" containerID="cri-o://c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef" gracePeriod=30 Oct 31 21:48:24.085894 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.085860 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:48:24.148538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.147443 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls\") pod \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " Oct 31 21:48:24.148538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.147509 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-kube-api-access-lw5zw\" (UniqueName: \"kubernetes.io/projected/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-kube-api-access-lw5zw\") pod \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " Oct 31 21:48:24.148538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.147605 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-openshift-service-ca\") pod \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " Oct 31 21:48:24.148538 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.147737 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-metrics-certs\") pod \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\" (UID: \"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5\") " Oct 31 21:48:24.150727 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.150694 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-openshift-service-ca" pod "6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" (UID: "6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:24.172905 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.172863 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls" pod "6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" (UID: "6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:48:24.192134 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.192070 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-kube-api-access-lw5zw" pod "6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" (UID: "6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5"). InnerVolumeSpecName "kube-api-access-lw5zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:24.224877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.215531 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-metrics-certs" pod "6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" (UID: "6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:48:24.249812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.249769 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:24.249812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.249816 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:24.250092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.249830 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw5zw\" (UniqueName: \"kubernetes.io/projected/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-kube-api-access-lw5zw\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:24.250092 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.249844 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:24.311548 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.311457 5188 generic.go:356] "Generic (PLEG): container finished" podID="6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" containerID="c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef" exitCode=0 Oct 31 21:48:24.311548 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.311508 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" Oct 31 21:48:24.311548 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.311539 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" event={"ID":"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5","Type":"ContainerDied","Data":"c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef"} Oct 31 21:48:24.311820 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.311612 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" event={"ID":"6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5","Type":"ContainerDied","Data":"93b992a312b7e60697b43e1e7a975d87e7f44fd3a72dfd2331ed9e3a03c8c781"} Oct 31 21:48:24.311820 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.311641 5188 scope.go:122] "RemoveContainer" containerID="c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef" Oct 31 21:48:24.324156 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.324129 5188 scope.go:122] "RemoveContainer" containerID="c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef" Oct 31 21:48:24.324555 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:48:24.324523 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef\": container with ID starting with c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef not found: ID does not exist" containerID="c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef" Oct 31 21:48:24.324666 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.324561 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef"} err="failed to get container status \"c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef\": rpc error: code = NotFound desc = could not find container \"c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef\": container with ID starting with c67feea07974e32cf4fb911a45afb81ecf28a219894855df8ca5d27548b4daef not found: ID does not exist" Oct 31 21:48:24.361675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.361634 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd"] Oct 31 21:48:24.365185 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:24.365146 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd"] Oct 31 21:48:25.474857 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:25.474817 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" path="/var/lib/kubelet/pods/6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5/volumes" Oct 31 21:48:27.008834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:27.008778 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b01ff6-6b84cf9b48-5m7wd" podUID="6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.218:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:48:27.558569 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:27.558522 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:27.571942 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:27.571913 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:28.356038 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:28.355997 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:48:32.547889 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:48:32.547850 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:48:35.318449 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.318408 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78"] Oct 31 21:48:35.319025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.319009 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" containerName="maas-api" Oct 31 21:48:35.319103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.319026 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" containerName="maas-api" Oct 31 21:48:35.319103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.319046 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6800f9ca-3103-4c97-82ae-ff7aaa223df3" containerName="payload-pre-processing" Oct 31 21:48:35.319103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.319052 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6800f9ca-3103-4c97-82ae-ff7aaa223df3" containerName="payload-pre-processing" Oct 31 21:48:35.319103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.319089 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9966a67d-a22f-451f-987a-448ca7b01aa9" containerName="payload-processing" Oct 31 21:48:35.319103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.319094 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9966a67d-a22f-451f-987a-448ca7b01aa9" containerName="payload-processing" Oct 31 21:48:35.319374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.319182 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9966a67d-a22f-451f-987a-448ca7b01aa9" containerName="payload-processing" Oct 31 21:48:35.319374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.319191 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a84b8e6-2b1f-478e-b4cb-f0b070fb3fd5" containerName="maas-api" Oct 31 21:48:35.319374 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.319199 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6800f9ca-3103-4c97-82ae-ff7aaa223df3" containerName="payload-pre-processing" Oct 31 21:48:35.323759 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.323733 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.361761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.348485 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-6b5dab-serving-cert\"" Oct 31 21:48:35.364780 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.363564 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78"] Oct 31 21:48:35.446816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.446775 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb5e084-18e0-47d2-85f7-a5102d5a8055-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.447021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.446840 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf829\" (UniqueName: \"kubernetes.io/projected/abb5e084-18e0-47d2-85f7-a5102d5a8055-kube-api-access-vf829\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.447021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.446959 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.447021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.447006 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.548527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.548480 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf829\" (UniqueName: \"kubernetes.io/projected/abb5e084-18e0-47d2-85f7-a5102d5a8055-kube-api-access-vf829\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.548527 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.548530 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.548768 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.548589 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.548811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.548790 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb5e084-18e0-47d2-85f7-a5102d5a8055-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.549540 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.549510 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb5e084-18e0-47d2-85f7-a5102d5a8055-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.551152 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.551126 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.551406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.551381 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.597181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.597065 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf829\" (UniqueName: \"kubernetes.io/projected/abb5e084-18e0-47d2-85f7-a5102d5a8055-kube-api-access-vf829\") pod \"maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.640387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.639756 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:35.958522 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:35.958483 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78"] Oct 31 21:48:35.959359 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:48:35.959289 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb5e084_18e0_47d2_85f7_a5102d5a8055.slice/crio-24fda4731b78eda20f90e2d9ba81b19a2a88eaf5ff49786c57e1ca45fa8e3fd8 WatchSource:0}: Error finding container 24fda4731b78eda20f90e2d9ba81b19a2a88eaf5ff49786c57e1ca45fa8e3fd8: Status 404 returned error can't find the container with id 24fda4731b78eda20f90e2d9ba81b19a2a88eaf5ff49786c57e1ca45fa8e3fd8 Oct 31 21:48:36.421245 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:36.421198 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" event={"ID":"abb5e084-18e0-47d2-85f7-a5102d5a8055","Type":"ContainerStarted","Data":"f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020"} Oct 31 21:48:36.421771 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:36.421289 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" event={"ID":"abb5e084-18e0-47d2-85f7-a5102d5a8055","Type":"ContainerStarted","Data":"24fda4731b78eda20f90e2d9ba81b19a2a88eaf5ff49786c57e1ca45fa8e3fd8"} Oct 31 21:48:36.423571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:36.422576 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:36.464959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:36.464885 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" podStartSLOduration=1.1474565509999999 podStartE2EDuration="1.464865097s" totalImagesPullingTime="317.408546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:36.464410602 +0000 UTC m=+1963.157367995" watchObservedRunningTime="2026-10-31 21:48:36.464865097 +0000 UTC m=+1963.157822491" Oct 31 21:48:40.499280 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.499240 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd"] Oct 31 21:48:40.504000 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.503968 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:40.523544 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.523503 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd"] Oct 31 21:48:40.527641 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.527603 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-fb32d4-dockercfg-kshq4\"" Oct 31 21:48:40.585181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.585139 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-fb32d4\"" Oct 31 21:48:40.645349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.632205 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nkm\" (UniqueName: \"kubernetes.io/projected/eb07ecde-e243-402e-965e-be6a87cc6d6a-kube-api-access-46nkm\") pod \"payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd\" (UID: \"eb07ecde-e243-402e-965e-be6a87cc6d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:40.645349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.632469 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb07ecde-e243-402e-965e-be6a87cc6d6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd\" (UID: \"eb07ecde-e243-402e-965e-be6a87cc6d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:40.736761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.736713 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nkm\" (UniqueName: \"kubernetes.io/projected/eb07ecde-e243-402e-965e-be6a87cc6d6a-kube-api-access-46nkm\") pod \"payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd\" (UID: \"eb07ecde-e243-402e-965e-be6a87cc6d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:40.737167 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.737139 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb07ecde-e243-402e-965e-be6a87cc6d6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd\" (UID: \"eb07ecde-e243-402e-965e-be6a87cc6d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:40.738387 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.738362 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb07ecde-e243-402e-965e-be6a87cc6d6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd\" (UID: \"eb07ecde-e243-402e-965e-be6a87cc6d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:40.760368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.760265 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nkm\" (UniqueName: \"kubernetes.io/projected/eb07ecde-e243-402e-965e-be6a87cc6d6a-kube-api-access-46nkm\") pod \"payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd\" (UID: \"eb07ecde-e243-402e-965e-be6a87cc6d6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:40.821560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:40.821511 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:41.131956 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:41.131918 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd"] Oct 31 21:48:41.133038 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:48:41.133004 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb07ecde_e243_402e_965e_be6a87cc6d6a.slice/crio-cd884cbfc6ed5e27f30e3517527531cdf4b44146d406fd786d9dce0c7f11b3ad WatchSource:0}: Error finding container cd884cbfc6ed5e27f30e3517527531cdf4b44146d406fd786d9dce0c7f11b3ad: Status 404 returned error can't find the container with id cd884cbfc6ed5e27f30e3517527531cdf4b44146d406fd786d9dce0c7f11b3ad Oct 31 21:48:41.462833 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:41.462736 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" event={"ID":"eb07ecde-e243-402e-965e-be6a87cc6d6a","Type":"ContainerStarted","Data":"0c0ccb7736cb0d82a741e9d6c3d544725614156a894ba7bc2370d872978799ed"} Oct 31 21:48:41.462833 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:41.462799 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" event={"ID":"eb07ecde-e243-402e-965e-be6a87cc6d6a","Type":"ContainerStarted","Data":"cd884cbfc6ed5e27f30e3517527531cdf4b44146d406fd786d9dce0c7f11b3ad"} Oct 31 21:48:41.463056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:41.462949 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:41.499031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:41.498977 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" podStartSLOduration=1.4989644819999999 podStartE2EDuration="1.498964482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:41.498235048 +0000 UTC m=+1968.191192442" watchObservedRunningTime="2026-10-31 21:48:41.498964482 +0000 UTC m=+1968.191921874" Oct 31 21:48:43.443923 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:43.443885 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 21:48:43.725225 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:43.725128 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl"] Oct 31 21:48:45.324448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.324391 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz"] Oct 31 21:48:45.331708 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.331673 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:45.346789 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.346755 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz"] Oct 31 21:48:45.349912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.349880 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-5237cd\"" Oct 31 21:48:45.350096 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.349992 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-5237cd-dockercfg-7vmtm\"" Oct 31 21:48:45.475349 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.474582 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n"] Oct 31 21:48:45.565025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.564984 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:45.579064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.579014 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n"] Oct 31 21:48:45.595272 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.595227 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdpt\" (UniqueName: \"kubernetes.io/projected/83480925-f78a-400c-83f5-6d44260c7d3c-kube-api-access-qmdpt\") pod \"payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz\" (UID: \"83480925-f78a-400c-83f5-6d44260c7d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:45.595497 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.595464 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83480925-f78a-400c-83f5-6d44260c7d3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz\" (UID: \"83480925-f78a-400c-83f5-6d44260c7d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:45.696595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.696549 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6668ee01-98ae-4bbc-b134-3df50eeeb23f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n\" (UID: \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:45.696801 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.696770 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdpt\" (UniqueName: \"kubernetes.io/projected/83480925-f78a-400c-83f5-6d44260c7d3c-kube-api-access-qmdpt\") pod \"payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz\" (UID: \"83480925-f78a-400c-83f5-6d44260c7d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:45.696955 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.696927 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcphq\" (UniqueName: \"kubernetes.io/projected/6668ee01-98ae-4bbc-b134-3df50eeeb23f-kube-api-access-kcphq\") pod \"payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n\" (UID: \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:45.697123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.696994 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83480925-f78a-400c-83f5-6d44260c7d3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz\" (UID: \"83480925-f78a-400c-83f5-6d44260c7d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:45.697781 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.697756 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83480925-f78a-400c-83f5-6d44260c7d3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz\" (UID: \"83480925-f78a-400c-83f5-6d44260c7d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:45.718767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.718734 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdpt\" (UniqueName: \"kubernetes.io/projected/83480925-f78a-400c-83f5-6d44260c7d3c-kube-api-access-qmdpt\") pod \"payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz\" (UID: \"83480925-f78a-400c-83f5-6d44260c7d3c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:45.798637 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.798580 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcphq\" (UniqueName: \"kubernetes.io/projected/6668ee01-98ae-4bbc-b134-3df50eeeb23f-kube-api-access-kcphq\") pod \"payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n\" (UID: \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:45.798861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.798725 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6668ee01-98ae-4bbc-b134-3df50eeeb23f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n\" (UID: \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:45.799526 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.799496 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6668ee01-98ae-4bbc-b134-3df50eeeb23f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n\" (UID: \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:45.820006 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.819969 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcphq\" (UniqueName: \"kubernetes.io/projected/6668ee01-98ae-4bbc-b134-3df50eeeb23f-kube-api-access-kcphq\") pod \"payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n\" (UID: \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:45.884842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.884783 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:45.951345 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:45.950495 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:46.204193 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.204157 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n"] Oct 31 21:48:46.206378 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:48:46.206346 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6668ee01_98ae_4bbc_b134_3df50eeeb23f.slice/crio-03b0c6c7a143138d66ee04f6aab6e71d0bf7ae95dc37188a123a37a40375ee02 WatchSource:0}: Error finding container 03b0c6c7a143138d66ee04f6aab6e71d0bf7ae95dc37188a123a37a40375ee02: Status 404 returned error can't find the container with id 03b0c6c7a143138d66ee04f6aab6e71d0bf7ae95dc37188a123a37a40375ee02 Oct 31 21:48:46.291485 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.291448 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz"] Oct 31 21:48:46.293759 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:48:46.293716 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83480925_f78a_400c_83f5_6d44260c7d3c.slice/crio-c0029698b8071ff799b6137d23f639069cd8ea87db0716a3c97aabcbb87c57e6 WatchSource:0}: Error finding container c0029698b8071ff799b6137d23f639069cd8ea87db0716a3c97aabcbb87c57e6: Status 404 returned error can't find the container with id c0029698b8071ff799b6137d23f639069cd8ea87db0716a3c97aabcbb87c57e6 Oct 31 21:48:46.519478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.519359 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" event={"ID":"83480925-f78a-400c-83f5-6d44260c7d3c","Type":"ContainerStarted","Data":"a8fc7227f12f63e601b188183b1c75936ead110da58ecd6611171362a6395084"} Oct 31 21:48:46.519478 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.519433 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" event={"ID":"83480925-f78a-400c-83f5-6d44260c7d3c","Type":"ContainerStarted","Data":"c0029698b8071ff799b6137d23f639069cd8ea87db0716a3c97aabcbb87c57e6"} Oct 31 21:48:46.520011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.519501 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:46.521455 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.521420 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" event={"ID":"6668ee01-98ae-4bbc-b134-3df50eeeb23f","Type":"ContainerStarted","Data":"463ded2a1846fde73af96e4f4732ba44d3f7ecff84e0ddee5d5f9e6946489c80"} Oct 31 21:48:46.521634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.521474 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" event={"ID":"6668ee01-98ae-4bbc-b134-3df50eeeb23f","Type":"ContainerStarted","Data":"03b0c6c7a143138d66ee04f6aab6e71d0bf7ae95dc37188a123a37a40375ee02"} Oct 31 21:48:46.521634 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.521516 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:46.559067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.558998 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" podStartSLOduration=1.558981969 podStartE2EDuration="1.558981969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:46.555607456 +0000 UTC m=+1973.248564846" watchObservedRunningTime="2026-10-31 21:48:46.558981969 +0000 UTC m=+1973.251939417" Oct 31 21:48:46.595015 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:46.594957 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" podStartSLOduration=1.5949431170000001 podStartE2EDuration="1.594943117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:46.591513282 +0000 UTC m=+1973.284470686" watchObservedRunningTime="2026-10-31 21:48:46.594943117 +0000 UTC m=+1973.287900510" Oct 31 21:48:48.725474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:48.725408 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" podUID="fa6222fe-f686-435c-bb08-c76a8d5de879" containerName="payload-processing" containerID="cri-o://33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158" gracePeriod=25 Oct 31 21:48:49.067037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.067007 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:48:49.143304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.143217 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6222fe-f686-435c-bb08-c76a8d5de879-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6222fe-f686-435c-bb08-c76a8d5de879-plugins-config-volume\") pod \"fa6222fe-f686-435c-bb08-c76a8d5de879\" (UID: \"fa6222fe-f686-435c-bb08-c76a8d5de879\") " Oct 31 21:48:49.143560 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.143375 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6222fe-f686-435c-bb08-c76a8d5de879-kube-api-access-qgtr4\" (UniqueName: \"kubernetes.io/projected/fa6222fe-f686-435c-bb08-c76a8d5de879-kube-api-access-qgtr4\") pod \"fa6222fe-f686-435c-bb08-c76a8d5de879\" (UID: \"fa6222fe-f686-435c-bb08-c76a8d5de879\") " Oct 31 21:48:49.144488 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.144449 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6222fe-f686-435c-bb08-c76a8d5de879-plugins-config-volume" pod "fa6222fe-f686-435c-bb08-c76a8d5de879" (UID: "fa6222fe-f686-435c-bb08-c76a8d5de879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:49.145295 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.145269 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6222fe-f686-435c-bb08-c76a8d5de879-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:49.146683 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.146651 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6222fe-f686-435c-bb08-c76a8d5de879-kube-api-access-qgtr4" pod "fa6222fe-f686-435c-bb08-c76a8d5de879" (UID: "fa6222fe-f686-435c-bb08-c76a8d5de879"). InnerVolumeSpecName "kube-api-access-qgtr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:49.246439 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.246406 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgtr4\" (UniqueName: \"kubernetes.io/projected/fa6222fe-f686-435c-bb08-c76a8d5de879-kube-api-access-qgtr4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:48:49.569080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.568083 5188 generic.go:356] "Generic (PLEG): container finished" podID="fa6222fe-f686-435c-bb08-c76a8d5de879" containerID="33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158" exitCode=0 Oct 31 21:48:49.569080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.568128 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" event={"ID":"fa6222fe-f686-435c-bb08-c76a8d5de879","Type":"ContainerDied","Data":"33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158"} Oct 31 21:48:49.569080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.568243 5188 scope.go:122] "RemoveContainer" containerID="33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158" Oct 31 21:48:49.569080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.568225 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" event={"ID":"fa6222fe-f686-435c-bb08-c76a8d5de879","Type":"ContainerDied","Data":"57bb1723c9b4ee90a8e275c1cfdc22702c7aac5abc30534f026cbded6115dfe6"} Oct 31 21:48:49.569080 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.568476 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl" Oct 31 21:48:49.591589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.588075 5188 scope.go:122] "RemoveContainer" containerID="33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158" Oct 31 21:48:49.591589 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:48:49.588545 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158\": container with ID starting with 33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158 not found: ID does not exist" containerID="33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158" Oct 31 21:48:49.591589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.588590 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158"} err="failed to get container status \"33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158\": rpc error: code = NotFound desc = could not find container \"33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158\": container with ID starting with 33a4c5e115cfa42b8fb5a79b9ba61a68ce333a6efc6616e5d095a0941f8aa158 not found: ID does not exist" Oct 31 21:48:49.617239 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.617167 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl"] Oct 31 21:48:49.626720 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:49.626677 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad121b-5c8c48fb87-m6zjl"] Oct 31 21:48:51.475551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:51.475513 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6222fe-f686-435c-bb08-c76a8d5de879" path="/var/lib/kubelet/pods/fa6222fe-f686-435c-bb08-c76a8d5de879/volumes" Oct 31 21:48:52.474085 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:52.474048 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 21:48:56.494559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:56.494514 5188 scope.go:122] "RemoveContainer" containerID="4dbcb05cd8cc89dc94e881bd776e67e4e87304871315936cc786b734166db598" Oct 31 21:48:57.532098 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:57.532063 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 21:48:57.532612 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:57.532219 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 21:48:59.077048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.077007 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n"] Oct 31 21:48:59.077949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.077920 5188 cpu_manager.go:414] "containerMap: removing container" podUID="fa6222fe-f686-435c-bb08-c76a8d5de879" containerName="payload-processing" Oct 31 21:48:59.077949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.077949 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6222fe-f686-435c-bb08-c76a8d5de879" containerName="payload-processing" Oct 31 21:48:59.078141 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.078106 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6222fe-f686-435c-bb08-c76a8d5de879" containerName="payload-processing" Oct 31 21:48:59.084100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.084072 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.113751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.113702 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-fb32d4-serving-cert\"" Oct 31 21:48:59.123946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.123909 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n"] Oct 31 21:48:59.227146 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.227107 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.227361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.227174 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkj8c\" (UniqueName: \"kubernetes.io/projected/df656d96-07dd-4660-a2dc-810e24f43929-kube-api-access-zkj8c\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.227361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.227238 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df656d96-07dd-4660-a2dc-810e24f43929-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.227361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.227282 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.328490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.328395 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.328791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.328766 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkj8c\" (UniqueName: \"kubernetes.io/projected/df656d96-07dd-4660-a2dc-810e24f43929-kube-api-access-zkj8c\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.328961 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.328924 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df656d96-07dd-4660-a2dc-810e24f43929-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.329142 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.329125 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.330304 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.330283 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df656d96-07dd-4660-a2dc-810e24f43929-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.343354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.338948 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.348352 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.345367 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.369393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.369336 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkj8c\" (UniqueName: \"kubernetes.io/projected/df656d96-07dd-4660-a2dc-810e24f43929-kube-api-access-zkj8c\") pod \"maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.404622 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.403701 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:48:59.719338 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:48:59.719281 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n"] Oct 31 21:48:59.722796 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:48:59.722766 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf656d96_07dd_4660_a2dc_810e24f43929.slice/crio-1db8050c0882af2096d75a7a1e212535c86f698408918ab317e64ffebea7b86c WatchSource:0}: Error finding container 1db8050c0882af2096d75a7a1e212535c86f698408918ab317e64ffebea7b86c: Status 404 returned error can't find the container with id 1db8050c0882af2096d75a7a1e212535c86f698408918ab317e64ffebea7b86c Oct 31 21:49:00.696896 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:00.696843 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" event={"ID":"df656d96-07dd-4660-a2dc-810e24f43929","Type":"ContainerStarted","Data":"a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99"} Oct 31 21:49:00.697454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:00.696914 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" event={"ID":"df656d96-07dd-4660-a2dc-810e24f43929","Type":"ContainerStarted","Data":"1db8050c0882af2096d75a7a1e212535c86f698408918ab317e64ffebea7b86c"} Oct 31 21:49:00.697454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:00.697067 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:49:00.740520 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:00.740462 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" podStartSLOduration=1.327142244 podStartE2EDuration="1.740444945s" totalImagesPullingTime="413.302701ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:49:00.73618403 +0000 UTC m=+1987.429141424" watchObservedRunningTime="2026-10-31 21:49:00.740444945 +0000 UTC m=+1987.433402338" Oct 31 21:49:02.540848 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:49:02.540803 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:49:04.351519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.351458 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n"] Oct 31 21:49:04.352747 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.351744 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" podUID="6cfd5de2-9903-47a4-b181-cd449243a179" containerName="maas-api" containerID="cri-o://50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab" gracePeriod=30 Oct 31 21:49:04.668928 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.668887 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:49:04.834722 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.834683 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cfd5de2-9903-47a4-b181-cd449243a179-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfd5de2-9903-47a4-b181-cd449243a179-openshift-service-ca\") pod \"6cfd5de2-9903-47a4-b181-cd449243a179\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " Oct 31 21:49:04.835203 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.835176 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cfd5de2-9903-47a4-b181-cd449243a179-kube-api-access-jdbsn\" (UniqueName: \"kubernetes.io/projected/6cfd5de2-9903-47a4-b181-cd449243a179-kube-api-access-jdbsn\") pod \"6cfd5de2-9903-47a4-b181-cd449243a179\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " Oct 31 21:49:04.835395 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.835376 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-metrics-certs\") pod \"6cfd5de2-9903-47a4-b181-cd449243a179\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " Oct 31 21:49:04.835559 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.835545 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls\") pod \"6cfd5de2-9903-47a4-b181-cd449243a179\" (UID: \"6cfd5de2-9903-47a4-b181-cd449243a179\") " Oct 31 21:49:04.838458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.838428 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cfd5de2-9903-47a4-b181-cd449243a179-openshift-service-ca" pod "6cfd5de2-9903-47a4-b181-cd449243a179" (UID: "6cfd5de2-9903-47a4-b181-cd449243a179"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:49:04.847625 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.847560 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls" pod "6cfd5de2-9903-47a4-b181-cd449243a179" (UID: "6cfd5de2-9903-47a4-b181-cd449243a179"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:49:04.850122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.850069 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-metrics-certs" pod "6cfd5de2-9903-47a4-b181-cd449243a179" (UID: "6cfd5de2-9903-47a4-b181-cd449243a179"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:49:04.861205 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.858096 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfd5de2-9903-47a4-b181-cd449243a179-kube-api-access-jdbsn" pod "6cfd5de2-9903-47a4-b181-cd449243a179" (UID: "6cfd5de2-9903-47a4-b181-cd449243a179"). InnerVolumeSpecName "kube-api-access-jdbsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:49:04.909444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.893628 5188 generic.go:356] "Generic (PLEG): container finished" podID="6cfd5de2-9903-47a4-b181-cd449243a179" containerID="50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab" exitCode=0 Oct 31 21:49:04.909444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.893710 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" event={"ID":"6cfd5de2-9903-47a4-b181-cd449243a179","Type":"ContainerDied","Data":"50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab"} Oct 31 21:49:04.909444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.893768 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" event={"ID":"6cfd5de2-9903-47a4-b181-cd449243a179","Type":"ContainerDied","Data":"bb474bf1a7c60a699cf7bab772b827f13c9590543266a1bf19958216284ef9b8"} Oct 31 21:49:04.909444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.893801 5188 scope.go:122] "RemoveContainer" containerID="50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab" Oct 31 21:49:04.909444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.894027 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n" Oct 31 21:49:04.924192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.924163 5188 scope.go:122] "RemoveContainer" containerID="50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab" Oct 31 21:49:04.924605 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:49:04.924578 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab\": container with ID starting with 50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab not found: ID does not exist" containerID="50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab" Oct 31 21:49:04.924678 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.924614 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab"} err="failed to get container status \"50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab\": rpc error: code = NotFound desc = could not find container \"50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab\": container with ID starting with 50eaf2874e3fa8c86ba8908b2c0fa661e79e97093a70812c2b6b66ef44e8a6ab not found: ID does not exist" Oct 31 21:49:04.937825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.937783 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cfd5de2-9903-47a4-b181-cd449243a179-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:49:04.937825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.937825 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbsn\" (UniqueName: \"kubernetes.io/projected/6cfd5de2-9903-47a4-b181-cd449243a179-kube-api-access-jdbsn\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:49:04.938019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.937843 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:49:04.938019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.937857 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6cfd5de2-9903-47a4-b181-cd449243a179-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:49:04.950282 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.950171 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n"] Oct 31 21:49:04.955566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:04.955531 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad121b-6f9cf87d7f-x8d5n"] Oct 31 21:49:05.197585 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.197500 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5"] Oct 31 21:49:05.198168 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.198151 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6cfd5de2-9903-47a4-b181-cd449243a179" containerName="maas-api" Oct 31 21:49:05.198267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.198168 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cfd5de2-9903-47a4-b181-cd449243a179" containerName="maas-api" Oct 31 21:49:05.198267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.198261 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cfd5de2-9903-47a4-b181-cd449243a179" containerName="maas-api" Oct 31 21:49:05.203396 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.203372 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.218839 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.218810 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-5237cd-serving-cert\"" Oct 31 21:49:05.223057 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.222779 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5"] Oct 31 21:49:05.344812 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.344774 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.345008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.344886 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8a256-3038-419a-b2df-fbf1991bfe30-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.345008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.344950 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.345008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.344981 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmfd\" (UniqueName: \"kubernetes.io/projected/17c8a256-3038-419a-b2df-fbf1991bfe30-kube-api-access-mxmfd\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.449693 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.449565 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.449693 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.449648 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmfd\" (UniqueName: \"kubernetes.io/projected/17c8a256-3038-419a-b2df-fbf1991bfe30-kube-api-access-mxmfd\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.450178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.449907 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.450178 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.449978 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8a256-3038-419a-b2df-fbf1991bfe30-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.450990 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.450961 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8a256-3038-419a-b2df-fbf1991bfe30-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.458606 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.458398 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.461180 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.461124 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.474891 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.474856 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmfd\" (UniqueName: \"kubernetes.io/projected/17c8a256-3038-419a-b2df-fbf1991bfe30-kube-api-access-mxmfd\") pod \"maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.484145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.484060 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cfd5de2-9903-47a4-b181-cd449243a179" path="/var/lib/kubelet/pods/6cfd5de2-9903-47a4-b181-cd449243a179/volumes" Oct 31 21:49:05.535978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.535936 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:05.876823 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:49:05.876787 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c8a256_3038_419a_b2df_fbf1991bfe30.slice/crio-0b7dffd7ed720a02c624705026169e88e121e37baf6df9203d9db2f6eca03ffc WatchSource:0}: Error finding container 0b7dffd7ed720a02c624705026169e88e121e37baf6df9203d9db2f6eca03ffc: Status 404 returned error can't find the container with id 0b7dffd7ed720a02c624705026169e88e121e37baf6df9203d9db2f6eca03ffc Oct 31 21:49:05.877611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.877574 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5"] Oct 31 21:49:05.905136 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:05.905088 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" event={"ID":"17c8a256-3038-419a-b2df-fbf1991bfe30","Type":"ContainerStarted","Data":"0b7dffd7ed720a02c624705026169e88e121e37baf6df9203d9db2f6eca03ffc"} Oct 31 21:49:06.715695 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:06.715639 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 21:49:06.928817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:06.928774 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" event={"ID":"17c8a256-3038-419a-b2df-fbf1991bfe30","Type":"ContainerStarted","Data":"4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493"} Oct 31 21:49:06.929053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:06.929007 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:06.987411 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:06.980372 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" podStartSLOduration=1.664259589 podStartE2EDuration="1.980351093s" totalImagesPullingTime="316.091504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:49:06.979779174 +0000 UTC m=+1993.672736559" watchObservedRunningTime="2026-10-31 21:49:06.980351093 +0000 UTC m=+1993.673308490" Oct 31 21:49:08.711784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.711738 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm"] Oct 31 21:49:08.716565 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.716535 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:08.742089 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.742054 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-44405d-dockercfg-74v5v\"" Oct 31 21:49:08.742353 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.742152 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-44405d\"" Oct 31 21:49:08.746767 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.746734 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm"] Oct 31 21:49:08.861401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.858866 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931df1cf-f6d0-4c1e-b87f-e596aba68266-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm\" (UID: \"931df1cf-f6d0-4c1e-b87f-e596aba68266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:08.861401 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.858985 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgq9\" (UniqueName: \"kubernetes.io/projected/931df1cf-f6d0-4c1e-b87f-e596aba68266-kube-api-access-vvgq9\") pod \"payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm\" (UID: \"931df1cf-f6d0-4c1e-b87f-e596aba68266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:08.961113 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.961065 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931df1cf-f6d0-4c1e-b87f-e596aba68266-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm\" (UID: \"931df1cf-f6d0-4c1e-b87f-e596aba68266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:08.961481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.961460 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgq9\" (UniqueName: \"kubernetes.io/projected/931df1cf-f6d0-4c1e-b87f-e596aba68266-kube-api-access-vvgq9\") pod \"payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm\" (UID: \"931df1cf-f6d0-4c1e-b87f-e596aba68266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:08.965075 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.965009 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931df1cf-f6d0-4c1e-b87f-e596aba68266-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm\" (UID: \"931df1cf-f6d0-4c1e-b87f-e596aba68266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:08.983969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:08.983926 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgq9\" (UniqueName: \"kubernetes.io/projected/931df1cf-f6d0-4c1e-b87f-e596aba68266-kube-api-access-vvgq9\") pod \"payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm\" (UID: \"931df1cf-f6d0-4c1e-b87f-e596aba68266\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:09.033556 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:09.033513 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:09.359635 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:09.359600 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm"] Oct 31 21:49:09.360182 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:49:09.360150 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod931df1cf_f6d0_4c1e_b87f_e596aba68266.slice/crio-2deec949dd04c484f65c9c1f53912ab3e5c20e32a09eb33365792e9055ddbdbd WatchSource:0}: Error finding container 2deec949dd04c484f65c9c1f53912ab3e5c20e32a09eb33365792e9055ddbdbd: Status 404 returned error can't find the container with id 2deec949dd04c484f65c9c1f53912ab3e5c20e32a09eb33365792e9055ddbdbd Oct 31 21:49:09.962753 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:09.962709 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" event={"ID":"931df1cf-f6d0-4c1e-b87f-e596aba68266","Type":"ContainerStarted","Data":"169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697"} Oct 31 21:49:09.963180 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:09.962763 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" event={"ID":"931df1cf-f6d0-4c1e-b87f-e596aba68266","Type":"ContainerStarted","Data":"2deec949dd04c484f65c9c1f53912ab3e5c20e32a09eb33365792e9055ddbdbd"} Oct 31 21:49:09.963180 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:09.962795 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:10.003346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:10.000393 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" podStartSLOduration=2.000373434 podStartE2EDuration="2.000373434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:49:09.998520968 +0000 UTC m=+1996.691478363" watchObservedRunningTime="2026-10-31 21:49:10.000373434 +0000 UTC m=+1996.693330828" Oct 31 21:49:12.942443 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:12.942411 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 21:49:20.975089 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:20.975055 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 21:49:28.568866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.568826 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc"] Oct 31 21:49:28.575419 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.575385 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.613949 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.613909 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-44405d-serving-cert\"" Oct 31 21:49:28.627823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.627763 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc"] Oct 31 21:49:28.666257 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.666207 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.666551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.666394 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5fb\" (UniqueName: \"kubernetes.io/projected/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-kube-api-access-gf5fb\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.666551 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.666522 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.666702 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.666591 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.782350 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.777217 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.788344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.773299 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.788344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.783738 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5fb\" (UniqueName: \"kubernetes.io/projected/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-kube-api-access-gf5fb\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.788344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.783899 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.788344 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.783982 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.811435 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.810281 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5fb\" (UniqueName: \"kubernetes.io/projected/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-kube-api-access-gf5fb\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.816818 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.815807 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.816818 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.816801 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:28.893249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:28.893199 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:29.213609 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:49:29.213564 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66030d3_d520_4c7c_b136_9ecdcdb6fd16.slice/crio-7afa622b9a2c4e17e4cb479856dc5884b0ba6e66d356d6c4ef5b40912247105f WatchSource:0}: Error finding container 7afa622b9a2c4e17e4cb479856dc5884b0ba6e66d356d6c4ef5b40912247105f: Status 404 returned error can't find the container with id 7afa622b9a2c4e17e4cb479856dc5884b0ba6e66d356d6c4ef5b40912247105f Oct 31 21:49:29.214791 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:29.214638 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc"] Oct 31 21:49:30.132762 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:30.132720 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" event={"ID":"e66030d3-d520-4c7c-b136-9ecdcdb6fd16","Type":"ContainerStarted","Data":"be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05"} Oct 31 21:49:30.133202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:30.132792 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:30.133202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:30.132810 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" event={"ID":"e66030d3-d520-4c7c-b136-9ecdcdb6fd16","Type":"ContainerStarted","Data":"7afa622b9a2c4e17e4cb479856dc5884b0ba6e66d356d6c4ef5b40912247105f"} Oct 31 21:49:30.178752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:30.178675 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" podStartSLOduration=1.8063937110000001 podStartE2EDuration="2.178651407s" totalImagesPullingTime="372.257696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:49:30.172225028 +0000 UTC m=+2016.865182422" watchObservedRunningTime="2026-10-31 21:49:30.178651407 +0000 UTC m=+2016.871608803" Oct 31 21:49:32.594442 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:49:32.594396 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:49:36.146488 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:36.146395 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 21:49:59.870154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:59.870111 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp"] Oct 31 21:49:59.877253 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:59.877222 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:49:59.899369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:59.899299 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp"] Oct 31 21:49:59.901978 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:59.901947 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-0877f5\"/\"default-dockercfg-757jp\"" Oct 31 21:49:59.902520 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:59.902355 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-0877f5\"/\"kube-root-ca.crt\"" Oct 31 21:49:59.902669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:59.902378 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-0877f5\"/\"openshift-service-ca.crt\"" Oct 31 21:49:59.902751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:49:59.902438 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-0877f5\"/\"auth-test-model-0877f5-kserve-self-signed-certs\"" Oct 31 21:50:00.039028 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.038990 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-home\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.039268 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.039061 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlpfv\" (UniqueName: \"kubernetes.io/projected/706c4b4a-08bd-4e0f-840d-2664e3b974d4-kube-api-access-mlpfv\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.039268 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.039236 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-dshm\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.039436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.039284 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tls-certs\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.039436 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.039424 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-model-cache\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.039533 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.039479 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tmp-dir\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.142403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.142144 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlpfv\" (UniqueName: \"kubernetes.io/projected/706c4b4a-08bd-4e0f-840d-2664e3b974d4-kube-api-access-mlpfv\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.142403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.142287 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-dshm\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.142403 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.142341 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tls-certs\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.142964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.142406 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-model-cache\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.142964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.142485 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tmp-dir\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.142964 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.142547 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-home\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.143280 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.143055 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-home\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.143784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.143742 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-model-cache\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.144453 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.144419 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tmp-dir\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.166237 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.161615 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-dshm\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.170386 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.167487 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlpfv\" (UniqueName: \"kubernetes.io/projected/706c4b4a-08bd-4e0f-840d-2664e3b974d4-kube-api-access-mlpfv\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.173115 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.173036 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tls-certs\") pod \"auth-test-model-0877f5-kserve-68f5b474c7-tp7gp\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.195529 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.195493 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:00.509210 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:00.509154 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp"] Oct 31 21:50:00.509747 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:50:00.509715 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod706c4b4a_08bd_4e0f_840d_2664e3b974d4.slice/crio-8ab2f0680f9b2bc1b824ebb3dc916bec634bbd9feaca4d64e7cc0201d6f0f3c4 WatchSource:0}: Error finding container 8ab2f0680f9b2bc1b824ebb3dc916bec634bbd9feaca4d64e7cc0201d6f0f3c4: Status 404 returned error can't find the container with id 8ab2f0680f9b2bc1b824ebb3dc916bec634bbd9feaca4d64e7cc0201d6f0f3c4 Oct 31 21:50:01.118595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.118552 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6"] Oct 31 21:50:01.122917 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.122887 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.145903 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.140567 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-efdee4-serving-cert\"" Oct 31 21:50:01.147260 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.147219 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6"] Oct 31 21:50:01.257811 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.257763 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961bc1bc-2807-405d-8c93-cf544bde3479-openshift-service-ca\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.258024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.257885 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-maas-api-tls\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.258024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.257921 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2gh5\" (UniqueName: \"kubernetes.io/projected/961bc1bc-2807-405d-8c93-cf544bde3479-kube-api-access-h2gh5\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.258024 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.257972 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-metrics-certs\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.359248 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.359206 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961bc1bc-2807-405d-8c93-cf544bde3479-openshift-service-ca\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.359503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.359273 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-maas-api-tls\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.359503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.359301 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2gh5\" (UniqueName: \"kubernetes.io/projected/961bc1bc-2807-405d-8c93-cf544bde3479-kube-api-access-h2gh5\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.359503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.359354 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-metrics-certs\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.361108 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.361067 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961bc1bc-2807-405d-8c93-cf544bde3479-openshift-service-ca\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.362869 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.362693 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-metrics-certs\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.362869 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.362840 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-maas-api-tls\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.431448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.417821 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" event={"ID":"706c4b4a-08bd-4e0f-840d-2664e3b974d4","Type":"ContainerStarted","Data":"fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287"} Oct 31 21:50:01.431448 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.417894 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" event={"ID":"706c4b4a-08bd-4e0f-840d-2664e3b974d4","Type":"ContainerStarted","Data":"8ab2f0680f9b2bc1b824ebb3dc916bec634bbd9feaca4d64e7cc0201d6f0f3c4"} Oct 31 21:50:01.449269 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.449223 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2gh5\" (UniqueName: \"kubernetes.io/projected/961bc1bc-2807-405d-8c93-cf544bde3479-kube-api-access-h2gh5\") pod \"maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:01.739336 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:01.739218 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:02.063364 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:02.063294 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" podStartSLOduration=3.06327426 podStartE2EDuration="3.06327426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:50:01.467860926 +0000 UTC m=+2048.160818319" watchObservedRunningTime="2026-10-31 21:50:02.06327426 +0000 UTC m=+2048.756231651" Oct 31 21:50:02.064003 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:02.063981 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6"] Oct 31 21:50:02.064820 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:50:02.064790 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961bc1bc_2807_405d_8c93_cf544bde3479.slice/crio-6af0c62699a329ff4c1002917654e30cc16fd1457d01679acec616df48bfce45 WatchSource:0}: Error finding container 6af0c62699a329ff4c1002917654e30cc16fd1457d01679acec616df48bfce45: Status 404 returned error can't find the container with id 6af0c62699a329ff4c1002917654e30cc16fd1457d01679acec616df48bfce45 Oct 31 21:50:02.196370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:02.196310 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:02.428834 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:02.428705 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" event={"ID":"961bc1bc-2807-405d-8c93-cf544bde3479","Type":"ContainerStarted","Data":"6af0c62699a329ff4c1002917654e30cc16fd1457d01679acec616df48bfce45"} Oct 31 21:50:02.540602 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:50:02.540560 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:50:03.445205 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:03.445158 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" event={"ID":"961bc1bc-2807-405d-8c93-cf544bde3479","Type":"ContainerStarted","Data":"9afd2823306790dc9cf656c93ccb1b273623b1312045866cf5d3cf5dc1446d08"} Oct 31 21:50:03.445751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:03.445271 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:03.488642 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:03.488575 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" podStartSLOduration=2.129910792 podStartE2EDuration="2.488559934s" totalImagesPullingTime="358.649142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:50:03.484956308 +0000 UTC m=+2050.177913700" watchObservedRunningTime="2026-10-31 21:50:03.488559934 +0000 UTC m=+2050.181517325" Oct 31 21:50:09.459703 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:09.459671 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:50:10.195989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:10.195944 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:10.209723 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:10.209684 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:10.514988 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:10.514896 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:50:32.539316 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:50:32.539269 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:50:33.302690 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.302643 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm"] Oct 31 21:50:33.308212 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.308181 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.328738 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.328700 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm"] Oct 31 21:50:33.330872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.330835 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-5237cd\"/\"openshift-service-ca.crt\"" Oct 31 21:50:33.330872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.330842 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-5237cd\"/\"kube-root-ca.crt\"" Oct 31 21:50:33.331137 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.331003 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-5237cd\"/\"default-dockercfg-pn6gf\"" Oct 31 21:50:33.331137 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.331090 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-5237cd\"/\"auth-test-model-5237cd-kserve-self-signed-certs\"" Oct 31 21:50:33.352530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.352192 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-dshm\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.352530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.352358 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tls-certs\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.352530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.352474 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s29t\" (UniqueName: \"kubernetes.io/projected/9d333e23-8c2f-44af-b5de-4ae19d6263e5-kube-api-access-7s29t\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.352980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.352546 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-home\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.352980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.352656 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tmp-dir\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.352980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.352769 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-model-cache\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.454659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.454615 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-model-cache\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.454887 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.454769 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-dshm\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.454887 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.454833 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tls-certs\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.455020 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.454925 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s29t\" (UniqueName: \"kubernetes.io/projected/9d333e23-8c2f-44af-b5de-4ae19d6263e5-kube-api-access-7s29t\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.455020 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.454995 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-home\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.455144 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.455083 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tmp-dir\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.455144 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.455129 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-model-cache\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.455615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.455581 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tmp-dir\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.455742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.455721 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-home\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.457428 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.457400 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-dshm\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.457832 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.457811 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tls-certs\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.476872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.476836 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s29t\" (UniqueName: \"kubernetes.io/projected/9d333e23-8c2f-44af-b5de-4ae19d6263e5-kube-api-access-7s29t\") pod \"auth-test-model-5237cd-kserve-68dbd64565-pbfwm\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.631908 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.631868 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:33.967494 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:33.967402 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm"] Oct 31 21:50:34.707214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:34.707168 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" event={"ID":"9d333e23-8c2f-44af-b5de-4ae19d6263e5","Type":"ContainerStarted","Data":"aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9"} Oct 31 21:50:34.707214 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:34.707222 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" event={"ID":"9d333e23-8c2f-44af-b5de-4ae19d6263e5","Type":"ContainerStarted","Data":"5e3c4ff137b50d193b0423ddfc4c33607ff6d7493ecf49e503aaa6aca4794db9"} Oct 31 21:50:34.749049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:34.748980 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" podStartSLOduration=1.7489616730000002 podStartE2EDuration="1.748961673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:50:34.746023517 +0000 UTC m=+2081.438980910" watchObservedRunningTime="2026-10-31 21:50:34.748961673 +0000 UTC m=+2081.441919065" Oct 31 21:50:35.632406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:35.632364 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:43.632750 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:43.632712 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:43.645517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:43.645483 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:43.788454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:43.788420 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:50:53.904493 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:53.904459 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:50:53.910068 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:50:53.910039 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:51:02.541543 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:51:02.541503 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:51:20.051676 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.051567 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6"] Oct 31 21:51:20.052795 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.051859 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" podUID="961bc1bc-2807-405d-8c93-cf544bde3479" containerName="maas-api" containerID="cri-o://9afd2823306790dc9cf656c93ccb1b273623b1312045866cf5d3cf5dc1446d08" gracePeriod=30 Oct 31 21:51:20.205578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.205540 5188 generic.go:356] "Generic (PLEG): container finished" podID="961bc1bc-2807-405d-8c93-cf544bde3479" containerID="9afd2823306790dc9cf656c93ccb1b273623b1312045866cf5d3cf5dc1446d08" exitCode=0 Oct 31 21:51:20.205830 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.205810 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" event={"ID":"961bc1bc-2807-405d-8c93-cf544bde3479","Type":"ContainerDied","Data":"9afd2823306790dc9cf656c93ccb1b273623b1312045866cf5d3cf5dc1446d08"} Oct 31 21:51:20.411398 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.410753 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:51:20.520348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.519714 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-metrics-certs\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-metrics-certs\") pod \"961bc1bc-2807-405d-8c93-cf544bde3479\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " Oct 31 21:51:20.520348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.519769 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961bc1bc-2807-405d-8c93-cf544bde3479-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961bc1bc-2807-405d-8c93-cf544bde3479-openshift-service-ca\") pod \"961bc1bc-2807-405d-8c93-cf544bde3479\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " Oct 31 21:51:20.520348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.519899 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-maas-api-tls\") pod \"961bc1bc-2807-405d-8c93-cf544bde3479\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " Oct 31 21:51:20.520348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.519945 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961bc1bc-2807-405d-8c93-cf544bde3479-kube-api-access-h2gh5\" (UniqueName: \"kubernetes.io/projected/961bc1bc-2807-405d-8c93-cf544bde3479-kube-api-access-h2gh5\") pod \"961bc1bc-2807-405d-8c93-cf544bde3479\" (UID: \"961bc1bc-2807-405d-8c93-cf544bde3479\") " Oct 31 21:51:20.527347 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.526310 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961bc1bc-2807-405d-8c93-cf544bde3479-openshift-service-ca" pod "961bc1bc-2807-405d-8c93-cf544bde3479" (UID: "961bc1bc-2807-405d-8c93-cf544bde3479"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:51:20.534348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.528565 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-metrics-certs" pod "961bc1bc-2807-405d-8c93-cf544bde3479" (UID: "961bc1bc-2807-405d-8c93-cf544bde3479"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:51:20.534348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.530142 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961bc1bc-2807-405d-8c93-cf544bde3479-kube-api-access-h2gh5" pod "961bc1bc-2807-405d-8c93-cf544bde3479" (UID: "961bc1bc-2807-405d-8c93-cf544bde3479"). InnerVolumeSpecName "kube-api-access-h2gh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:51:20.534348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.530231 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-maas-api-tls" pod "961bc1bc-2807-405d-8c93-cf544bde3479" (UID: "961bc1bc-2807-405d-8c93-cf544bde3479"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:51:20.627348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.626700 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:51:20.627348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.626744 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/961bc1bc-2807-405d-8c93-cf544bde3479-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:51:20.627348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.626760 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/961bc1bc-2807-405d-8c93-cf544bde3479-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:51:20.627348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:20.626773 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2gh5\" (UniqueName: \"kubernetes.io/projected/961bc1bc-2807-405d-8c93-cf544bde3479-kube-api-access-h2gh5\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:51:21.215385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:21.215310 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" Oct 31 21:51:21.215880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:21.215313 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6" event={"ID":"961bc1bc-2807-405d-8c93-cf544bde3479","Type":"ContainerDied","Data":"6af0c62699a329ff4c1002917654e30cc16fd1457d01679acec616df48bfce45"} Oct 31 21:51:21.215880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:21.215495 5188 scope.go:122] "RemoveContainer" containerID="9afd2823306790dc9cf656c93ccb1b273623b1312045866cf5d3cf5dc1446d08" Oct 31 21:51:21.262661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:21.262623 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6"] Oct 31 21:51:21.272756 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:21.272704 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-efdee4-7fdcb89ddc-z6vf6"] Oct 31 21:51:21.475656 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:21.475558 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961bc1bc-2807-405d-8c93-cf544bde3479" path="/var/lib/kubelet/pods/961bc1bc-2807-405d-8c93-cf544bde3479/volumes" Oct 31 21:51:23.036188 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:23.036127 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk"] Oct 31 21:51:23.044966 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:23.044927 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a059aa-l88tk"] Oct 31 21:51:23.474576 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:51:23.474534 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f653e756-ced4-46ec-9100-b4f81380b05a" path="/var/lib/kubelet/pods/f653e756-ced4-46ec-9100-b4f81380b05a/volumes" Oct 31 21:51:32.540150 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:51:32.540106 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:52:02.539150 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:52:02.539107 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:52:14.009969 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:14.009912 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p"] Oct 31 21:52:23.031667 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:23.031628 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn"] Oct 31 21:52:23.041005 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:23.040967 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e338f6f-7vfwn"] Oct 31 21:52:23.477009 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:23.476964 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b5e1fe-ced7-426c-b422-b36d27089aa6" path="/var/lib/kubelet/pods/73b5e1fe-ced7-426c-b422-b36d27089aa6/volumes" Oct 31 21:52:29.040145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.040097 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" podUID="bbaebe34-779f-4ea3-b254-51511cb1c455" containerName="main" containerID="cri-o://16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940" gracePeriod=45 Oct 31 21:52:29.314829 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.314795 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:52:29.414847 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.414807 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbaebe34-779f-4ea3-b254-51511cb1c455-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaebe34-779f-4ea3-b254-51511cb1c455-tls-certs\") pod \"bbaebe34-779f-4ea3-b254-51511cb1c455\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " Oct 31 21:52:29.415097 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.415076 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbaebe34-779f-4ea3-b254-51511cb1c455-kube-api-access-rrq29\" (UniqueName: \"kubernetes.io/projected/bbaebe34-779f-4ea3-b254-51511cb1c455-kube-api-access-rrq29\") pod \"bbaebe34-779f-4ea3-b254-51511cb1c455\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " Oct 31 21:52:29.415159 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.415138 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-tmp-dir\") pod \"bbaebe34-779f-4ea3-b254-51511cb1c455\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " Oct 31 21:52:29.415218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.415190 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-model-cache\") pod \"bbaebe34-779f-4ea3-b254-51511cb1c455\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " Oct 31 21:52:29.415218 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.415210 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-home\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-home\") pod \"bbaebe34-779f-4ea3-b254-51511cb1c455\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " Oct 31 21:52:29.415333 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.415234 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-dshm\") pod \"bbaebe34-779f-4ea3-b254-51511cb1c455\" (UID: \"bbaebe34-779f-4ea3-b254-51511cb1c455\") " Oct 31 21:52:29.415884 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.415798 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-model-cache" pod "bbaebe34-779f-4ea3-b254-51511cb1c455" (UID: "bbaebe34-779f-4ea3-b254-51511cb1c455"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:52:29.415884 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.415785 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-tmp-dir" pod "bbaebe34-779f-4ea3-b254-51511cb1c455" (UID: "bbaebe34-779f-4ea3-b254-51511cb1c455"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:52:29.415884 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.415850 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-home" pod "bbaebe34-779f-4ea3-b254-51511cb1c455" (UID: "bbaebe34-779f-4ea3-b254-51511cb1c455"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:52:29.417419 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.417392 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbaebe34-779f-4ea3-b254-51511cb1c455-kube-api-access-rrq29" pod "bbaebe34-779f-4ea3-b254-51511cb1c455" (UID: "bbaebe34-779f-4ea3-b254-51511cb1c455"). InnerVolumeSpecName "kube-api-access-rrq29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:52:29.417557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.417539 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbaebe34-779f-4ea3-b254-51511cb1c455-tls-certs" pod "bbaebe34-779f-4ea3-b254-51511cb1c455" (UID: "bbaebe34-779f-4ea3-b254-51511cb1c455"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:52:29.417659 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.417631 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-dshm" pod "bbaebe34-779f-4ea3-b254-51511cb1c455" (UID: "bbaebe34-779f-4ea3-b254-51511cb1c455"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:52:29.517963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.517926 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:52:29.517963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.517964 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbaebe34-779f-4ea3-b254-51511cb1c455-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:52:29.518249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.517981 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrq29\" (UniqueName: \"kubernetes.io/projected/bbaebe34-779f-4ea3-b254-51511cb1c455-kube-api-access-rrq29\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:52:29.518249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.517994 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:52:29.518249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.518009 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:52:29.518249 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.518021 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbaebe34-779f-4ea3-b254-51511cb1c455-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:52:29.821285 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.821247 5188 generic.go:356] "Generic (PLEG): container finished" podID="bbaebe34-779f-4ea3-b254-51511cb1c455" containerID="16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940" exitCode=0 Oct 31 21:52:29.821537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.821356 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" Oct 31 21:52:29.821537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.821372 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" event={"ID":"bbaebe34-779f-4ea3-b254-51511cb1c455","Type":"ContainerDied","Data":"16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940"} Oct 31 21:52:29.821537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.821425 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p" event={"ID":"bbaebe34-779f-4ea3-b254-51511cb1c455","Type":"ContainerDied","Data":"f62ba0648dc80b33915fa9045fe3cd577a1c172918df341d04bbe187933fbdad"} Oct 31 21:52:29.821537 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.821450 5188 scope.go:122] "RemoveContainer" containerID="16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940" Oct 31 21:52:29.832878 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.832850 5188 scope.go:122] "RemoveContainer" containerID="16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940" Oct 31 21:52:29.833205 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:52:29.833186 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940\": container with ID starting with 16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940 not found: ID does not exist" containerID="16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940" Oct 31 21:52:29.833271 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.833214 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940"} err="failed to get container status \"16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940\": rpc error: code = NotFound desc = could not find container \"16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940\": container with ID starting with 16167b47759e68f572e875ddd801ca27c1354ff7523ce8679336302b120b1940 not found: ID does not exist" Oct 31 21:52:29.882192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.881591 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p"] Oct 31 21:52:29.891782 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:29.891666 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c744c0/auth-test-model-c744c0-kserve-ddf7887cc-72w7p"] Oct 31 21:52:31.476728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:31.476682 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbaebe34-779f-4ea3-b254-51511cb1c455" path="/var/lib/kubelet/pods/bbaebe34-779f-4ea3-b254-51511cb1c455/volumes" Oct 31 21:52:32.538271 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:52:32.538228 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:52:54.497635 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.497591 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd"] Oct 31 21:52:54.498264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.498249 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bbaebe34-779f-4ea3-b254-51511cb1c455" containerName="main" Oct 31 21:52:54.498311 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.498265 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbaebe34-779f-4ea3-b254-51511cb1c455" containerName="main" Oct 31 21:52:54.498311 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.498304 5188 cpu_manager.go:414] "containerMap: removing container" podUID="961bc1bc-2807-405d-8c93-cf544bde3479" containerName="maas-api" Oct 31 21:52:54.498311 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.498309 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961bc1bc-2807-405d-8c93-cf544bde3479" containerName="maas-api" Oct 31 21:52:54.498464 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.498449 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbaebe34-779f-4ea3-b254-51511cb1c455" containerName="main" Oct 31 21:52:54.498503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.498468 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="961bc1bc-2807-405d-8c93-cf544bde3479" containerName="maas-api" Oct 31 21:52:54.502250 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.502226 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.516275 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.516247 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-4bd51b-serving-cert\"" Oct 31 21:52:54.524230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.524193 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.524404 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.524315 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.524510 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.524491 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crlzj\" (UniqueName: \"kubernetes.io/projected/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-kube-api-access-crlzj\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.524662 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.524641 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-metrics-certs\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.529921 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.529891 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd"] Oct 31 21:52:54.626534 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.626497 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crlzj\" (UniqueName: \"kubernetes.io/projected/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-kube-api-access-crlzj\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.626742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.626603 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-metrics-certs\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.626742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.626685 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.626742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.626717 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.627457 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.627429 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.629202 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.629179 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-metrics-certs\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.629305 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.629226 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.648495 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.648469 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crlzj\" (UniqueName: \"kubernetes.io/projected/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-kube-api-access-crlzj\") pod \"maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:54.822764 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:54.822679 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:55.142104 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:55.142072 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd"] Oct 31 21:52:55.144671 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:52:55.144639 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab1c8d8_de2b_4020_bde4_d190e8494c7f.slice/crio-8683b25367607c7244c8e11ea74cf2024ef1a6bd3ea91ffb982078c7f36da8e6 WatchSource:0}: Error finding container 8683b25367607c7244c8e11ea74cf2024ef1a6bd3ea91ffb982078c7f36da8e6: Status 404 returned error can't find the container with id 8683b25367607c7244c8e11ea74cf2024ef1a6bd3ea91ffb982078c7f36da8e6 Oct 31 21:52:55.146120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:55.146097 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:52:56.117840 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:56.117793 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" event={"ID":"2ab1c8d8-de2b-4020-bde4-d190e8494c7f","Type":"ContainerStarted","Data":"6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212"} Oct 31 21:52:56.118418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:56.117869 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" event={"ID":"2ab1c8d8-de2b-4020-bde4-d190e8494c7f","Type":"ContainerStarted","Data":"8683b25367607c7244c8e11ea74cf2024ef1a6bd3ea91ffb982078c7f36da8e6"} Oct 31 21:52:56.118418 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:56.118146 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:52:56.157727 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:56.157658 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" podStartSLOduration=1.582486917 podStartE2EDuration="2.157641509s" totalImagesPullingTime="575.154592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:52:56.155924592 +0000 UTC m=+2222.848881986" watchObservedRunningTime="2026-10-31 21:52:56.157641509 +0000 UTC m=+2222.850598904" Oct 31 21:52:56.622027 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:52:56.621991 5188 scope.go:122] "RemoveContainer" containerID="8b51ef9ad5695274c41d45bc99e49f3a17805f3f47f3308a9ac971304a459dee" Oct 31 21:53:02.131855 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:53:02.131823 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 21:53:02.539943 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:53:02.539847 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:53:32.541658 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:53:32.541611 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:53:56.656016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:53:56.655979 5188 scope.go:122] "RemoveContainer" containerID="3683845c0a943ff2916ba990b3189ab4a6bd4f9eaea5013133aa4fba2474b674" Oct 31 21:54:02.538925 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:54:02.538885 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:54:09.760021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:09.759932 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55"] Oct 31 21:54:09.760701 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:09.760177 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" containerID="cri-o://159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff" gracePeriod=30 Oct 31 21:54:15.079208 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.079178 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:54:15.224481 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224447 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-envoy\") pod \"fef870d9-ae63-4135-8a64-b4adab4ec07c\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " Oct 31 21:54:15.224669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224550 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-certs\") pod \"fef870d9-ae63-4135-8a64-b4adab4ec07c\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " Oct 31 21:54:15.224669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224578 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef870d9-ae63-4135-8a64-b4adab4ec07c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef870d9-ae63-4135-8a64-b4adab4ec07c-istiod-ca-cert\") pod \"fef870d9-ae63-4135-8a64-b4adab4ec07c\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " Oct 31 21:54:15.224669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224599 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-credential-socket\") pod \"fef870d9-ae63-4135-8a64-b4adab4ec07c\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " Oct 31 21:54:15.224669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224623 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-podinfo\") pod \"fef870d9-ae63-4135-8a64-b4adab4ec07c\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " Oct 31 21:54:15.224669 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224643 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-token\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-token\") pod \"fef870d9-ae63-4135-8a64-b4adab4ec07c\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " Oct 31 21:54:15.224954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224782 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-socket\") pod \"fef870d9-ae63-4135-8a64-b4adab4ec07c\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " Oct 31 21:54:15.224954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224871 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-data\") pod \"fef870d9-ae63-4135-8a64-b4adab4ec07c\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " Oct 31 21:54:15.224954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224853 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-certs" pod "fef870d9-ae63-4135-8a64-b4adab4ec07c" (UID: "fef870d9-ae63-4135-8a64-b4adab4ec07c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:15.224954 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224911 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-kube-api-access-lgjw9\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-kube-api-access-lgjw9\") pod \"fef870d9-ae63-4135-8a64-b4adab4ec07c\" (UID: \"fef870d9-ae63-4135-8a64-b4adab4ec07c\") " Oct 31 21:54:15.225197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.224954 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-credential-socket" pod "fef870d9-ae63-4135-8a64-b4adab4ec07c" (UID: "fef870d9-ae63-4135-8a64-b4adab4ec07c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:15.225197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.225057 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef870d9-ae63-4135-8a64-b4adab4ec07c-istiod-ca-cert" pod "fef870d9-ae63-4135-8a64-b4adab4ec07c" (UID: "fef870d9-ae63-4135-8a64-b4adab4ec07c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:15.225197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.225171 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-socket" pod "fef870d9-ae63-4135-8a64-b4adab4ec07c" (UID: "fef870d9-ae63-4135-8a64-b4adab4ec07c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:15.225197 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.225189 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-data" pod "fef870d9-ae63-4135-8a64-b4adab4ec07c" (UID: "fef870d9-ae63-4135-8a64-b4adab4ec07c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:15.226189 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.226156 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:15.226370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.226192 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:15.226370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.226209 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:15.226370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.226223 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef870d9-ae63-4135-8a64-b4adab4ec07c-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:15.226370 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.226236 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:15.227270 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.227241 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-podinfo" pod "fef870d9-ae63-4135-8a64-b4adab4ec07c" (UID: "fef870d9-ae63-4135-8a64-b4adab4ec07c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:54:15.227270 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.227259 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-envoy" pod "fef870d9-ae63-4135-8a64-b4adab4ec07c" (UID: "fef870d9-ae63-4135-8a64-b4adab4ec07c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:15.227482 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.227381 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-kube-api-access-lgjw9" pod "fef870d9-ae63-4135-8a64-b4adab4ec07c" (UID: "fef870d9-ae63-4135-8a64-b4adab4ec07c"). InnerVolumeSpecName "kube-api-access-lgjw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:15.227530 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.227485 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-token" pod "fef870d9-ae63-4135-8a64-b4adab4ec07c" (UID: "fef870d9-ae63-4135-8a64-b4adab4ec07c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:15.327302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.327202 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgjw9\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-kube-api-access-lgjw9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:15.327302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.327242 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:15.327302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.327252 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:15.327302 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.327260 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef870d9-ae63-4135-8a64-b4adab4ec07c-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:15.804389 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.804340 5188 generic.go:356] "Generic (PLEG): container finished" podID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerID="159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff" exitCode=0 Oct 31 21:54:15.804620 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.804392 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" Oct 31 21:54:15.804620 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.804438 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" event={"ID":"fef870d9-ae63-4135-8a64-b4adab4ec07c","Type":"ContainerDied","Data":"159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff"} Oct 31 21:54:15.804620 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.804506 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55" event={"ID":"fef870d9-ae63-4135-8a64-b4adab4ec07c","Type":"ContainerDied","Data":"00eaa6c5cab6dfe8528866f7e7ff92e0c30a130ad5e2f7d0b10cce923e4bb069"} Oct 31 21:54:15.804620 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.804537 5188 scope.go:122] "RemoveContainer" containerID="159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff" Oct 31 21:54:15.821042 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.821013 5188 scope.go:122] "RemoveContainer" containerID="159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff" Oct 31 21:54:15.821936 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:54:15.821906 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff\": container with ID starting with 159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff not found: ID does not exist" containerID="159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff" Oct 31 21:54:15.822066 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.821943 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff"} err="failed to get container status \"159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff\": rpc error: code = NotFound desc = could not find container \"159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff\": container with ID starting with 159b6f214a98497f8cd5fbe42f039d7e277dcb3d93262279ddea704e052f05ff not found: ID does not exist" Oct 31 21:54:15.871021 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.870941 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55"] Oct 31 21:54:15.881187 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:15.881148 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-thq55"] Oct 31 21:54:17.475744 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:17.475704 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" path="/var/lib/kubelet/pods/fef870d9-ae63-4135-8a64-b4adab4ec07c/volumes" Oct 31 21:54:19.638503 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.638459 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m"] Oct 31 21:54:19.639354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.639311 5188 cpu_manager.go:414] "containerMap: removing container" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" Oct 31 21:54:19.639354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.639353 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" Oct 31 21:54:19.639578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.639529 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef870d9-ae63-4135-8a64-b4adab4ec07c" containerName="istio-proxy" Oct 31 21:54:19.642904 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.642850 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.676382 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.676339 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m"] Oct 31 21:54:19.683245 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.683208 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-metrics-certs\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.683470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.683315 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca6993-280b-4f66-b320-c2d9dca8a873-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.683470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.683413 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.683470 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.683439 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlz9\" (UniqueName: \"kubernetes.io/projected/52ca6993-280b-4f66-b320-c2d9dca8a873-kube-api-access-mtlz9\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.701424 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.701388 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-56a6d4-serving-cert\"" Oct 31 21:54:19.784740 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.784694 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-metrics-certs\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.784967 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.784823 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca6993-280b-4f66-b320-c2d9dca8a873-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.784967 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.784867 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.784967 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.784894 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlz9\" (UniqueName: \"kubernetes.io/projected/52ca6993-280b-4f66-b320-c2d9dca8a873-kube-api-access-mtlz9\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.785266 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:54:19.785249 5188 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-serving-cert: secret "maas-api-e2e-extmod-b-56a6d4-serving-cert" not found Oct 31 21:54:19.785364 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:54:19.785349 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls podName:52ca6993-280b-4f66-b320-c2d9dca8a873 nodeName:}" failed. No retries permitted until 2026-10-31 21:54:20.285309879 +0000 UTC m=+2306.978267254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls") pod "maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" (UID: "52ca6993-280b-4f66-b320-c2d9dca8a873") : secret "maas-api-e2e-extmod-b-56a6d4-serving-cert" not found Oct 31 21:54:19.785845 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.785815 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca6993-280b-4f66-b320-c2d9dca8a873-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.788043 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.788014 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-metrics-certs\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:19.831937 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:19.831895 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlz9\" (UniqueName: \"kubernetes.io/projected/52ca6993-280b-4f66-b320-c2d9dca8a873-kube-api-access-mtlz9\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:20.291607 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:20.291558 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:20.294513 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:20.294470 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls\") pod \"maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:20.560784 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:20.560681 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:20.900489 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:54:20.900456 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ca6993_280b_4f66_b320_c2d9dca8a873.slice/crio-815976be28c085bee082c668c113b3223408b458f55dfebab08870e88b67dda0 WatchSource:0}: Error finding container 815976be28c085bee082c668c113b3223408b458f55dfebab08870e88b67dda0: Status 404 returned error can't find the container with id 815976be28c085bee082c668c113b3223408b458f55dfebab08870e88b67dda0 Oct 31 21:54:20.900837 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:20.900623 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m"] Oct 31 21:54:21.889011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:21.888969 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" event={"ID":"52ca6993-280b-4f66-b320-c2d9dca8a873","Type":"ContainerStarted","Data":"a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011"} Oct 31 21:54:21.889230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:21.889041 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" event={"ID":"52ca6993-280b-4f66-b320-c2d9dca8a873","Type":"ContainerStarted","Data":"815976be28c085bee082c668c113b3223408b458f55dfebab08870e88b67dda0"} Oct 31 21:54:21.889230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:21.889072 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:21.930866 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:21.930808 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" podStartSLOduration=2.5455582039999998 podStartE2EDuration="2.930791697s" totalImagesPullingTime="385.233493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:21.927710858 +0000 UTC m=+2308.620668250" watchObservedRunningTime="2026-10-31 21:54:21.930791697 +0000 UTC m=+2308.623749090" Oct 31 21:54:27.904226 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:27.904186 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 21:54:32.539624 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:54:32.539580 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:54:33.188154 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:33.188109 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78"] Oct 31 21:54:33.188484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:33.188354 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" containerID="cri-o://2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c" gracePeriod=30 Oct 31 21:54:38.575714 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.575685 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:54:38.588524 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588481 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-credential-socket\") pod \"45a58da4-2103-41cf-9493-604d51c5723c\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " Oct 31 21:54:38.588823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588560 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-istio-token\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-istio-token\") pod \"45a58da4-2103-41cf-9493-604d51c5723c\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " Oct 31 21:54:38.588823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588591 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-envoy\") pod \"45a58da4-2103-41cf-9493-604d51c5723c\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " Oct 31 21:54:38.588823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588622 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-certs\") pod \"45a58da4-2103-41cf-9493-604d51c5723c\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " Oct 31 21:54:38.588823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588650 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-socket\") pod \"45a58da4-2103-41cf-9493-604d51c5723c\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " Oct 31 21:54:38.588823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588683 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a58da4-2103-41cf-9493-604d51c5723c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a58da4-2103-41cf-9493-604d51c5723c-istiod-ca-cert\") pod \"45a58da4-2103-41cf-9493-604d51c5723c\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " Oct 31 21:54:38.588823 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588733 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-kube-api-access-k546t\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-kube-api-access-k546t\") pod \"45a58da4-2103-41cf-9493-604d51c5723c\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588832 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-credential-socket" pod "45a58da4-2103-41cf-9493-604d51c5723c" (UID: "45a58da4-2103-41cf-9493-604d51c5723c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588866 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45a58da4-2103-41cf-9493-604d51c5723c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a58da4-2103-41cf-9493-604d51c5723c-istio-podinfo\") pod \"45a58da4-2103-41cf-9493-604d51c5723c\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.588892 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-data\") pod \"45a58da4-2103-41cf-9493-604d51c5723c\" (UID: \"45a58da4-2103-41cf-9493-604d51c5723c\") " Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.589223 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a58da4-2103-41cf-9493-604d51c5723c-istiod-ca-cert" pod "45a58da4-2103-41cf-9493-604d51c5723c" (UID: "45a58da4-2103-41cf-9493-604d51c5723c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.589224 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-certs" pod "45a58da4-2103-41cf-9493-604d51c5723c" (UID: "45a58da4-2103-41cf-9493-604d51c5723c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.589465 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-data" pod "45a58da4-2103-41cf-9493-604d51c5723c" (UID: "45a58da4-2103-41cf-9493-604d51c5723c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.589558 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-socket" pod "45a58da4-2103-41cf-9493-604d51c5723c" (UID: "45a58da4-2103-41cf-9493-604d51c5723c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.589641 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.589662 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.589680 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:38.589851 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.589694 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a58da4-2103-41cf-9493-604d51c5723c-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:38.591490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.591443 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-envoy" pod "45a58da4-2103-41cf-9493-604d51c5723c" (UID: "45a58da4-2103-41cf-9493-604d51c5723c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:38.591490 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.591445 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45a58da4-2103-41cf-9493-604d51c5723c-istio-podinfo" pod "45a58da4-2103-41cf-9493-604d51c5723c" (UID: "45a58da4-2103-41cf-9493-604d51c5723c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:54:38.591687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.591642 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-istio-token" pod "45a58da4-2103-41cf-9493-604d51c5723c" (UID: "45a58da4-2103-41cf-9493-604d51c5723c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:38.591687 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.591664 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-kube-api-access-k546t" pod "45a58da4-2103-41cf-9493-604d51c5723c" (UID: "45a58da4-2103-41cf-9493-604d51c5723c"). InnerVolumeSpecName "kube-api-access-k546t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:38.691483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.691447 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:38.691483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.691475 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:38.691483 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.691484 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a58da4-2103-41cf-9493-604d51c5723c-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:38.691757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.691492 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k546t\" (UniqueName: \"kubernetes.io/projected/45a58da4-2103-41cf-9493-604d51c5723c-kube-api-access-k546t\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:38.691757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:38.691505 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a58da4-2103-41cf-9493-604d51c5723c-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:39.028151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.028114 5188 generic.go:356] "Generic (PLEG): container finished" podID="45a58da4-2103-41cf-9493-604d51c5723c" containerID="2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c" exitCode=0 Oct 31 21:54:39.028444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.028170 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" Oct 31 21:54:39.028444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.028204 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" event={"ID":"45a58da4-2103-41cf-9493-604d51c5723c","Type":"ContainerDied","Data":"2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c"} Oct 31 21:54:39.028444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.028307 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78" event={"ID":"45a58da4-2103-41cf-9493-604d51c5723c","Type":"ContainerDied","Data":"9342697c4c03dd277e2e7debade78e185b9a9f55784fe90ad6c902038d575f2e"} Oct 31 21:54:39.028444 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.028357 5188 scope.go:122] "RemoveContainer" containerID="2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c" Oct 31 21:54:39.042554 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.042519 5188 scope.go:122] "RemoveContainer" containerID="2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c" Oct 31 21:54:39.043178 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:54:39.043132 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c\": container with ID starting with 2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c not found: ID does not exist" containerID="2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c" Oct 31 21:54:39.043281 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.043185 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c"} err="failed to get container status \"2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c\": rpc error: code = NotFound desc = could not find container \"2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c\": container with ID starting with 2fb623d8e79d8e9cb11877b6920e5448a7dc6cb6fa85e0eed92647269feaa80c not found: ID does not exist" Oct 31 21:54:39.079773 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.079743 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78"] Oct 31 21:54:39.088755 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.088717 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-6nw78"] Oct 31 21:54:39.474377 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.474342 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a58da4-2103-41cf-9493-604d51c5723c" path="/var/lib/kubelet/pods/45a58da4-2103-41cf-9493-604d51c5723c/volumes" Oct 31 21:54:39.748813 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.748715 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx"] Oct 31 21:54:39.749383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:39.748989 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" containerID="cri-o://5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3" gracePeriod=30 Oct 31 21:54:42.825190 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:42.825151 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp"] Oct 31 21:54:45.032751 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.032666 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:54:45.068484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.068451 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5"] Oct 31 21:54:45.069313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.069282 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" Oct 31 21:54:45.069313 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.069307 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" Oct 31 21:54:45.069489 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.069388 5188 cpu_manager.go:414] "containerMap: removing container" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" Oct 31 21:54:45.069489 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.069397 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" Oct 31 21:54:45.069590 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.069574 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a58da4-2103-41cf-9493-604d51c5723c" containerName="istio-proxy" Oct 31 21:54:45.069648 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.069603 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerName="istio-proxy" Oct 31 21:54:45.073233 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.073207 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.094355 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.094059 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-696f9a-serving-cert\"" Oct 31 21:54:45.096407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.095448 5188 generic.go:356] "Generic (PLEG): container finished" podID="b314b11e-bba8-4bb2-978c-a53331086fa3" containerID="5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3" exitCode=0 Oct 31 21:54:45.096407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.095490 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" event={"ID":"b314b11e-bba8-4bb2-978c-a53331086fa3","Type":"ContainerDied","Data":"5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3"} Oct 31 21:54:45.096407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.095536 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" event={"ID":"b314b11e-bba8-4bb2-978c-a53331086fa3","Type":"ContainerDied","Data":"c054baaf548c8f84e84897f69f6fb61ca4b22660fb0697f0f2f4683ab385deec"} Oct 31 21:54:45.096407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.095571 5188 scope.go:122] "RemoveContainer" containerID="5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3" Oct 31 21:54:45.096407 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.095716 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx" Oct 31 21:54:45.105459 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.105427 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-credential-socket\") pod \"b314b11e-bba8-4bb2-978c-a53331086fa3\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " Oct 31 21:54:45.105742 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.105713 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-data\") pod \"b314b11e-bba8-4bb2-978c-a53331086fa3\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " Oct 31 21:54:45.105945 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.105925 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-token\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-token\") pod \"b314b11e-bba8-4bb2-978c-a53331086fa3\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " Oct 31 21:54:45.106237 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.106220 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-certs\") pod \"b314b11e-bba8-4bb2-978c-a53331086fa3\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " Oct 31 21:54:45.106432 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.106401 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-kube-api-access-hhlh8\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-kube-api-access-hhlh8\") pod \"b314b11e-bba8-4bb2-978c-a53331086fa3\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " Oct 31 21:54:45.106588 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.106571 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-socket\") pod \"b314b11e-bba8-4bb2-978c-a53331086fa3\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " Oct 31 21:54:45.107645 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.107622 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-podinfo\") pod \"b314b11e-bba8-4bb2-978c-a53331086fa3\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " Oct 31 21:54:45.107842 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.107815 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-envoy\") pod \"b314b11e-bba8-4bb2-978c-a53331086fa3\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " Oct 31 21:54:45.108076 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.108057 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b314b11e-bba8-4bb2-978c-a53331086fa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b314b11e-bba8-4bb2-978c-a53331086fa3-istiod-ca-cert\") pod \"b314b11e-bba8-4bb2-978c-a53331086fa3\" (UID: \"b314b11e-bba8-4bb2-978c-a53331086fa3\") " Oct 31 21:54:45.108775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.107685 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-certs" pod "b314b11e-bba8-4bb2-978c-a53331086fa3" (UID: "b314b11e-bba8-4bb2-978c-a53331086fa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:45.108775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.108068 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-credential-socket" pod "b314b11e-bba8-4bb2-978c-a53331086fa3" (UID: "b314b11e-bba8-4bb2-978c-a53331086fa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:45.108775 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.108248 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-data" pod "b314b11e-bba8-4bb2-978c-a53331086fa3" (UID: "b314b11e-bba8-4bb2-978c-a53331086fa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:45.109013 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.108938 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-socket" pod "b314b11e-bba8-4bb2-978c-a53331086fa3" (UID: "b314b11e-bba8-4bb2-978c-a53331086fa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:45.109877 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.109852 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b314b11e-bba8-4bb2-978c-a53331086fa3-istiod-ca-cert" pod "b314b11e-bba8-4bb2-978c-a53331086fa3" (UID: "b314b11e-bba8-4bb2-978c-a53331086fa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:45.110117 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.110093 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:45.110211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.110124 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:45.110211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.110137 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b314b11e-bba8-4bb2-978c-a53331086fa3-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:45.110211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.110152 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:45.110211 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.110167 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:45.118103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.115747 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5"] Oct 31 21:54:45.118103 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.116848 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-token" pod "b314b11e-bba8-4bb2-978c-a53331086fa3" (UID: "b314b11e-bba8-4bb2-978c-a53331086fa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:45.121572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.120164 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-envoy" pod "b314b11e-bba8-4bb2-978c-a53331086fa3" (UID: "b314b11e-bba8-4bb2-978c-a53331086fa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:45.121572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.120267 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-podinfo" pod "b314b11e-bba8-4bb2-978c-a53331086fa3" (UID: "b314b11e-bba8-4bb2-978c-a53331086fa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:54:45.126899 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.126856 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-kube-api-access-hhlh8" pod "b314b11e-bba8-4bb2-978c-a53331086fa3" (UID: "b314b11e-bba8-4bb2-978c-a53331086fa3"). InnerVolumeSpecName "kube-api-access-hhlh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:45.169151 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.169114 5188 scope.go:122] "RemoveContainer" containerID="5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3" Oct 31 21:54:45.169510 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:54:45.169477 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3\": container with ID starting with 5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3 not found: ID does not exist" containerID="5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3" Oct 31 21:54:45.169645 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.169520 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3"} err="failed to get container status \"5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3\": rpc error: code = NotFound desc = could not find container \"5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3\": container with ID starting with 5a3f13e529b052ba6ab07225cca3993c8bf02b5f45e4048746b5b789e05b20b3 not found: ID does not exist" Oct 31 21:54:45.211888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.211847 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9428ee1e-11af-405f-a792-0971229ef18c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.211888 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.211899 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.212169 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.212018 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwm2q\" (UniqueName: \"kubernetes.io/projected/9428ee1e-11af-405f-a792-0971229ef18c-kube-api-access-wwm2q\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.212220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.212194 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-metrics-certs\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.212367 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.212352 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:45.212417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.212371 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhlh8\" (UniqueName: \"kubernetes.io/projected/b314b11e-bba8-4bb2-978c-a53331086fa3-kube-api-access-hhlh8\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:45.212417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.212382 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:45.212417 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.212391 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b314b11e-bba8-4bb2-978c-a53331086fa3-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:45.313464 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.313364 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-metrics-certs\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.313640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.313481 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9428ee1e-11af-405f-a792-0971229ef18c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.313640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.313511 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.313640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.313571 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwm2q\" (UniqueName: \"kubernetes.io/projected/9428ee1e-11af-405f-a792-0971229ef18c-kube-api-access-wwm2q\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.314487 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.314456 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9428ee1e-11af-405f-a792-0971229ef18c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.316070 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.316047 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-metrics-certs\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.316264 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.316246 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.335393 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.335349 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwm2q\" (UniqueName: \"kubernetes.io/projected/9428ee1e-11af-405f-a792-0971229ef18c-kube-api-access-wwm2q\") pod \"maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.391980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.391942 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:45.448334 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.448263 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx"] Oct 31 21:54:45.454611 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.453838 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-294zx"] Oct 31 21:54:45.480372 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.479280 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b314b11e-bba8-4bb2-978c-a53331086fa3" path="/var/lib/kubelet/pods/b314b11e-bba8-4bb2-978c-a53331086fa3/volumes" Oct 31 21:54:45.709267 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:54:45.709229 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9428ee1e_11af_405f_a792_0971229ef18c.slice/crio-7ffd0ec198840d405f2dbe6251cc5838a22decd2e970e7116578c027fcfddb22 WatchSource:0}: Error finding container 7ffd0ec198840d405f2dbe6251cc5838a22decd2e970e7116578c027fcfddb22: Status 404 returned error can't find the container with id 7ffd0ec198840d405f2dbe6251cc5838a22decd2e970e7116578c027fcfddb22 Oct 31 21:54:45.710378 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:45.710349 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5"] Oct 31 21:54:46.110608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:46.106802 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" event={"ID":"9428ee1e-11af-405f-a792-0971229ef18c","Type":"ContainerStarted","Data":"7ffd0ec198840d405f2dbe6251cc5838a22decd2e970e7116578c027fcfddb22"} Oct 31 21:54:47.116997 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:47.116936 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" event={"ID":"9428ee1e-11af-405f-a792-0971229ef18c","Type":"ContainerStarted","Data":"0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792"} Oct 31 21:54:47.117505 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:47.117023 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:47.157665 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:47.157580 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" podStartSLOduration=1.834624323 podStartE2EDuration="2.157560128s" totalImagesPullingTime="322.935805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:47.155784856 +0000 UTC m=+2333.848742250" watchObservedRunningTime="2026-10-31 21:54:47.157560128 +0000 UTC m=+2333.850517520" Oct 31 21:54:53.131968 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:53.131930 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 21:54:53.431754 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:53.431663 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg"] Oct 31 21:54:53.432010 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:53.431886 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" containerID="cri-o://9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802" gracePeriod=30 Oct 31 21:54:57.849519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:57.849475 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" podUID="706c4b4a-08bd-4e0f-840d-2664e3b974d4" containerName="main" containerID="cri-o://fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287" gracePeriod=45 Oct 31 21:54:58.157769 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.157739 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:54:58.220122 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.220080 5188 generic.go:356] "Generic (PLEG): container finished" podID="706c4b4a-08bd-4e0f-840d-2664e3b974d4" containerID="fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287" exitCode=0 Oct 31 21:54:58.220368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.220140 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" event={"ID":"706c4b4a-08bd-4e0f-840d-2664e3b974d4","Type":"ContainerDied","Data":"fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287"} Oct 31 21:54:58.220368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.220170 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" event={"ID":"706c4b4a-08bd-4e0f-840d-2664e3b974d4","Type":"ContainerDied","Data":"8ab2f0680f9b2bc1b824ebb3dc916bec634bbd9feaca4d64e7cc0201d6f0f3c4"} Oct 31 21:54:58.220368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.220186 5188 scope.go:122] "RemoveContainer" containerID="fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287" Oct 31 21:54:58.220368 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.220364 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp" Oct 31 21:54:58.232880 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.232848 5188 scope.go:122] "RemoveContainer" containerID="fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287" Oct 31 21:54:58.233283 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:54:58.233257 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287\": container with ID starting with fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287 not found: ID does not exist" containerID="fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287" Oct 31 21:54:58.233406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.233290 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287"} err="failed to get container status \"fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287\": rpc error: code = NotFound desc = could not find container \"fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287\": container with ID starting with fd70358a09d7c46bfc7a26ad6f0f88117066142f356fd53bc8688460d94ad287 not found: ID does not exist" Oct 31 21:54:58.286920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.286862 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706c4b4a-08bd-4e0f-840d-2664e3b974d4-kube-api-access-mlpfv\" (UniqueName: \"kubernetes.io/projected/706c4b4a-08bd-4e0f-840d-2664e3b974d4-kube-api-access-mlpfv\") pod \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " Oct 31 21:54:58.287120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.286955 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-home\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-home\") pod \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " Oct 31 21:54:58.287120 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.287075 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tmp-dir\") pod \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " Oct 31 21:54:58.287262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.287132 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-model-cache\") pod \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " Oct 31 21:54:58.287262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.287246 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-dshm\") pod \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " Oct 31 21:54:58.287406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.287277 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-home" pod "706c4b4a-08bd-4e0f-840d-2664e3b974d4" (UID: "706c4b4a-08bd-4e0f-840d-2664e3b974d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:58.287406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.287318 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tls-certs\") pod \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\" (UID: \"706c4b4a-08bd-4e0f-840d-2664e3b974d4\") " Oct 31 21:54:58.287406 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.287315 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tmp-dir" pod "706c4b4a-08bd-4e0f-840d-2664e3b974d4" (UID: "706c4b4a-08bd-4e0f-840d-2664e3b974d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:58.287589 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.287485 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-model-cache" pod "706c4b4a-08bd-4e0f-840d-2664e3b974d4" (UID: "706c4b4a-08bd-4e0f-840d-2664e3b974d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:58.288008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.287977 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.288008 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.288003 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.288199 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.288016 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.289803 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.289762 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-dshm" pod "706c4b4a-08bd-4e0f-840d-2664e3b974d4" (UID: "706c4b4a-08bd-4e0f-840d-2664e3b974d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:58.289941 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.289796 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706c4b4a-08bd-4e0f-840d-2664e3b974d4-kube-api-access-mlpfv" pod "706c4b4a-08bd-4e0f-840d-2664e3b974d4" (UID: "706c4b4a-08bd-4e0f-840d-2664e3b974d4"). InnerVolumeSpecName "kube-api-access-mlpfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:58.290246 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.290227 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tls-certs" pod "706c4b4a-08bd-4e0f-840d-2664e3b974d4" (UID: "706c4b4a-08bd-4e0f-840d-2664e3b974d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:54:58.389317 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.389234 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706c4b4a-08bd-4e0f-840d-2664e3b974d4-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.389317 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.389272 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706c4b4a-08bd-4e0f-840d-2664e3b974d4-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.389317 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.389286 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlpfv\" (UniqueName: \"kubernetes.io/projected/706c4b4a-08bd-4e0f-840d-2664e3b974d4-kube-api-access-mlpfv\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.565064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.565030 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp"] Oct 31 21:54:58.574415 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.574383 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-0877f5/auth-test-model-0877f5-kserve-68f5b474c7-tp7gp"] Oct 31 21:54:58.703186 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.703161 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:54:58.794822 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.794788 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-socket\") pod \"a8828d49-b461-46a8-919e-39731dc06b1f\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " Oct 31 21:54:58.794822 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.794833 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-credential-socket\") pod \"a8828d49-b461-46a8-919e-39731dc06b1f\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " Oct 31 21:54:58.795111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.794900 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8828d49-b461-46a8-919e-39731dc06b1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8828d49-b461-46a8-919e-39731dc06b1f-istio-podinfo\") pod \"a8828d49-b461-46a8-919e-39731dc06b1f\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " Oct 31 21:54:58.795111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.794983 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-data\") pod \"a8828d49-b461-46a8-919e-39731dc06b1f\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " Oct 31 21:54:58.795111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795052 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8828d49-b461-46a8-919e-39731dc06b1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8828d49-b461-46a8-919e-39731dc06b1f-istiod-ca-cert\") pod \"a8828d49-b461-46a8-919e-39731dc06b1f\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " Oct 31 21:54:58.795111 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795098 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-istio-token\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-istio-token\") pod \"a8828d49-b461-46a8-919e-39731dc06b1f\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " Oct 31 21:54:58.795394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795129 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-envoy\") pod \"a8828d49-b461-46a8-919e-39731dc06b1f\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " Oct 31 21:54:58.795394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795139 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-socket" pod "a8828d49-b461-46a8-919e-39731dc06b1f" (UID: "a8828d49-b461-46a8-919e-39731dc06b1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:58.795394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795157 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-credential-socket" pod "a8828d49-b461-46a8-919e-39731dc06b1f" (UID: "a8828d49-b461-46a8-919e-39731dc06b1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:58.795394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795165 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-kube-api-access-9dcxw\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-kube-api-access-9dcxw\") pod \"a8828d49-b461-46a8-919e-39731dc06b1f\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " Oct 31 21:54:58.795394 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795205 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-certs\") pod \"a8828d49-b461-46a8-919e-39731dc06b1f\" (UID: \"a8828d49-b461-46a8-919e-39731dc06b1f\") " Oct 31 21:54:58.795699 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795504 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8828d49-b461-46a8-919e-39731dc06b1f-istiod-ca-cert" pod "a8828d49-b461-46a8-919e-39731dc06b1f" (UID: "a8828d49-b461-46a8-919e-39731dc06b1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:58.795761 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795694 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-data" pod "a8828d49-b461-46a8-919e-39731dc06b1f" (UID: "a8828d49-b461-46a8-919e-39731dc06b1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:58.795817 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795758 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-certs" pod "a8828d49-b461-46a8-919e-39731dc06b1f" (UID: "a8828d49-b461-46a8-919e-39731dc06b1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:58.795974 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795956 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.796049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795979 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.796049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.795993 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.796049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.796006 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8828d49-b461-46a8-919e-39731dc06b1f-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.796049 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.796020 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.797850 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.797821 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-istio-token" pod "a8828d49-b461-46a8-919e-39731dc06b1f" (UID: "a8828d49-b461-46a8-919e-39731dc06b1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:58.798116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.798090 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-kube-api-access-9dcxw" pod "a8828d49-b461-46a8-919e-39731dc06b1f" (UID: "a8828d49-b461-46a8-919e-39731dc06b1f"). InnerVolumeSpecName "kube-api-access-9dcxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:58.798116 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.798089 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-envoy" pod "a8828d49-b461-46a8-919e-39731dc06b1f" (UID: "a8828d49-b461-46a8-919e-39731dc06b1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:58.799918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.799890 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8828d49-b461-46a8-919e-39731dc06b1f-istio-podinfo" pod "a8828d49-b461-46a8-919e-39731dc06b1f" (UID: "a8828d49-b461-46a8-919e-39731dc06b1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:54:58.897517 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.897485 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.898034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.897539 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8828d49-b461-46a8-919e-39731dc06b1f-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.898034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.897550 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dcxw\" (UniqueName: \"kubernetes.io/projected/a8828d49-b461-46a8-919e-39731dc06b1f-kube-api-access-9dcxw\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:58.898034 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:58.897561 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8828d49-b461-46a8-919e-39731dc06b1f-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:54:59.229032 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.228994 5188 generic.go:356] "Generic (PLEG): container finished" podID="a8828d49-b461-46a8-919e-39731dc06b1f" containerID="9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802" exitCode=0 Oct 31 21:54:59.229242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.229060 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" Oct 31 21:54:59.229242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.229085 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" event={"ID":"a8828d49-b461-46a8-919e-39731dc06b1f","Type":"ContainerDied","Data":"9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802"} Oct 31 21:54:59.229242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.229150 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg" event={"ID":"a8828d49-b461-46a8-919e-39731dc06b1f","Type":"ContainerDied","Data":"de071b13d46a0a71c38ffd68b4a0921419919522ce8c924e7a652a99f73c022e"} Oct 31 21:54:59.229242 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.229179 5188 scope.go:122] "RemoveContainer" containerID="9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802" Oct 31 21:54:59.239660 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.239640 5188 scope.go:122] "RemoveContainer" containerID="9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802" Oct 31 21:54:59.239971 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:54:59.239951 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802\": container with ID starting with 9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802 not found: ID does not exist" containerID="9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802" Oct 31 21:54:59.240053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.239983 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802"} err="failed to get container status \"9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802\": rpc error: code = NotFound desc = could not find container \"9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802\": container with ID starting with 9906e996a12e6ad44c4bfd73984d7f98690ef085f6f6cc710f7d91ec46226802 not found: ID does not exist" Oct 31 21:54:59.276636 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.276578 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg"] Oct 31 21:54:59.286037 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.286005 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-5t6hg"] Oct 31 21:54:59.474333 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.474282 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706c4b4a-08bd-4e0f-840d-2664e3b974d4" path="/var/lib/kubelet/pods/706c4b4a-08bd-4e0f-840d-2664e3b974d4/volumes" Oct 31 21:54:59.474867 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:54:59.474851 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" path="/var/lib/kubelet/pods/a8828d49-b461-46a8-919e-39731dc06b1f/volumes" Oct 31 21:55:01.924814 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:01.924772 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm"] Oct 31 21:55:02.538688 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:55:02.538651 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:55:02.832959 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.832886 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp"] Oct 31 21:55:02.833514 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.833500 5188 cpu_manager.go:414] "containerMap: removing container" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" Oct 31 21:55:02.833583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.833516 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" Oct 31 21:55:02.833583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.833550 5188 cpu_manager.go:414] "containerMap: removing container" podUID="706c4b4a-08bd-4e0f-840d-2664e3b974d4" containerName="main" Oct 31 21:55:02.833583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.833555 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706c4b4a-08bd-4e0f-840d-2664e3b974d4" containerName="main" Oct 31 21:55:02.833680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.833661 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8828d49-b461-46a8-919e-39731dc06b1f" containerName="istio-proxy" Oct 31 21:55:02.833680 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.833674 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="706c4b4a-08bd-4e0f-840d-2664e3b974d4" containerName="main" Oct 31 21:55:02.838807 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.838778 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.854376 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.854349 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-44405d-openshift-default-dockercfg-vjr7p\"" Oct 31 21:55:02.855895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.855867 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-token\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.856018 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.855910 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff66395-e99b-411d-a3c5-f397a62f5004-istiod-ca-cert\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.856084 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.856063 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sn2l\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-kube-api-access-6sn2l\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.856123 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.856093 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-envoy\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.856184 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.856169 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-podinfo\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.856237 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.856225 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-certs\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.856278 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.856256 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.856399 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.856379 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-data\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.856460 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.856438 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-credential-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.868684 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.868659 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp"] Oct 31 21:55:02.957163 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957129 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-token\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.957698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957174 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff66395-e99b-411d-a3c5-f397a62f5004-istiod-ca-cert\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.957698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957258 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sn2l\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-kube-api-access-6sn2l\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.957698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957279 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-envoy\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.957698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957317 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-podinfo\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.957698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957403 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-certs\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.957698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957438 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.957698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957477 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-data\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.957698 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957492 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-credential-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.958145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957846 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-data\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.958145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957926 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.958145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.957951 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-credential-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.958145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.958014 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-certs\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.958145 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.958088 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff66395-e99b-411d-a3c5-f397a62f5004-istiod-ca-cert\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.959875 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.959851 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-envoy\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.960036 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.960017 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-podinfo\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.981843 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.981802 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-token\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:02.983545 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:02.983512 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sn2l\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-kube-api-access-6sn2l\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:03.153019 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:03.152983 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:03.542101 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:03.542068 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp"] Oct 31 21:55:03.542672 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:55:03.542644 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff66395_e99b_411d_a3c5_f397a62f5004.slice/crio-f22c639aee82852c249ccf0604b8f9b57c48129a7a1d1c32d562f6618847b877 WatchSource:0}: Error finding container f22c639aee82852c249ccf0604b8f9b57c48129a7a1d1c32d562f6618847b877: Status 404 returned error can't find the container with id f22c639aee82852c249ccf0604b8f9b57c48129a7a1d1c32d562f6618847b877 Oct 31 21:55:03.546088 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:03.546054 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:55:03.546171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:03.546122 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:55:03.546171 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:03.546152 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 21:55:04.279363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:04.279285 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" event={"ID":"0ff66395-e99b-411d-a3c5-f397a62f5004","Type":"ContainerStarted","Data":"f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975"} Oct 31 21:55:04.279803 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:04.279399 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" event={"ID":"0ff66395-e99b-411d-a3c5-f397a62f5004","Type":"ContainerStarted","Data":"f22c639aee82852c249ccf0604b8f9b57c48129a7a1d1c32d562f6618847b877"} Oct 31 21:55:04.373031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:04.372964 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" podStartSLOduration=2.372941162 podStartE2EDuration="2.372941162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:55:04.367913996 +0000 UTC m=+2351.060871385" watchObservedRunningTime="2026-10-31 21:55:04.372941162 +0000 UTC m=+2351.065898555" Oct 31 21:55:05.153361 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:05.153280 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:05.155447 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:05.155410 5188 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 21:55:05.155598 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:05.155481 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 21:55:06.153579 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:06.153539 5188 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 21:55:06.153998 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:06.153607 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 21:55:07.153608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:07.153564 5188 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 21:55:07.154048 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:07.153627 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 21:55:08.153717 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:08.153651 5188 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 21:55:08.154229 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:08.153731 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 21:55:09.153628 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:09.153585 5188 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 21:55:09.153853 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:09.153646 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 21:55:10.153477 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:10.153431 5188 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Oct 31 21:55:10.153679 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:10.153514 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Oct 31 21:55:11.159595 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:11.159554 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:11.160078 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:11.159845 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:11.160640 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:11.160622 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 21:55:16.948181 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:16.948134 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" podUID="9d333e23-8c2f-44af-b5de-4ae19d6263e5" containerName="main" containerID="cri-o://aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9" gracePeriod=45 Oct 31 21:55:17.258090 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.258058 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:55:17.418255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.418214 5188 generic.go:356] "Generic (PLEG): container finished" podID="9d333e23-8c2f-44af-b5de-4ae19d6263e5" containerID="aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9" exitCode=0 Oct 31 21:55:17.418255 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.418267 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" Oct 31 21:55:17.418571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.418306 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" event={"ID":"9d333e23-8c2f-44af-b5de-4ae19d6263e5","Type":"ContainerDied","Data":"aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9"} Oct 31 21:55:17.418571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.418369 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm" event={"ID":"9d333e23-8c2f-44af-b5de-4ae19d6263e5","Type":"ContainerDied","Data":"5e3c4ff137b50d193b0423ddfc4c33607ff6d7493ecf49e503aaa6aca4794db9"} Oct 31 21:55:17.418571 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.418387 5188 scope.go:122] "RemoveContainer" containerID="aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9" Oct 31 21:55:17.429282 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.429257 5188 scope.go:122] "RemoveContainer" containerID="aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9" Oct 31 21:55:17.429685 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:55:17.429660 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9\": container with ID starting with aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9 not found: ID does not exist" containerID="aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9" Oct 31 21:55:17.429787 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.429699 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9"} err="failed to get container status \"aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9\": rpc error: code = NotFound desc = could not find container \"aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9\": container with ID starting with aa9230485d054ff706f134f0dfda6f09a02c106789dcc7d4e9f229ae7b294cb9 not found: ID does not exist" Oct 31 21:55:17.439531 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.439467 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-model-cache\") pod \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " Oct 31 21:55:17.439870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.439552 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d333e23-8c2f-44af-b5de-4ae19d6263e5-kube-api-access-7s29t\" (UniqueName: \"kubernetes.io/projected/9d333e23-8c2f-44af-b5de-4ae19d6263e5-kube-api-access-7s29t\") pod \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " Oct 31 21:55:17.439870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.439731 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-home\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-home\") pod \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " Oct 31 21:55:17.439989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.439875 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tls-certs\") pod \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " Oct 31 21:55:17.439989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.439950 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-model-cache" pod "9d333e23-8c2f-44af-b5de-4ae19d6263e5" (UID: "9d333e23-8c2f-44af-b5de-4ae19d6263e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:17.439989 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.439968 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-dshm\") pod \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " Oct 31 21:55:17.440519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.440006 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-home" pod "9d333e23-8c2f-44af-b5de-4ae19d6263e5" (UID: "9d333e23-8c2f-44af-b5de-4ae19d6263e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:17.440519 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.440027 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tmp-dir\") pod \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\" (UID: \"9d333e23-8c2f-44af-b5de-4ae19d6263e5\") " Oct 31 21:55:17.440757 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.440733 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tmp-dir" pod "9d333e23-8c2f-44af-b5de-4ae19d6263e5" (UID: "9d333e23-8c2f-44af-b5de-4ae19d6263e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:17.441112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.441090 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:55:17.441112 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.441113 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:55:17.441308 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.441154 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:55:17.443539 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.443501 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-dshm" pod "9d333e23-8c2f-44af-b5de-4ae19d6263e5" (UID: "9d333e23-8c2f-44af-b5de-4ae19d6263e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:17.443539 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.443521 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d333e23-8c2f-44af-b5de-4ae19d6263e5-kube-api-access-7s29t" pod "9d333e23-8c2f-44af-b5de-4ae19d6263e5" (UID: "9d333e23-8c2f-44af-b5de-4ae19d6263e5"). InnerVolumeSpecName "kube-api-access-7s29t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:55:17.447461 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.447430 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tls-certs" pod "9d333e23-8c2f-44af-b5de-4ae19d6263e5" (UID: "9d333e23-8c2f-44af-b5de-4ae19d6263e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:55:17.542139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.542099 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s29t\" (UniqueName: \"kubernetes.io/projected/9d333e23-8c2f-44af-b5de-4ae19d6263e5-kube-api-access-7s29t\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:55:17.542139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.542134 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d333e23-8c2f-44af-b5de-4ae19d6263e5-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:55:17.542139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.542146 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d333e23-8c2f-44af-b5de-4ae19d6263e5-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:55:17.755799 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.755711 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm"] Oct 31 21:55:17.759383 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:17.759349 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-5237cd/auth-test-model-5237cd-kserve-68dbd64565-pbfwm"] Oct 31 21:55:19.476334 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:19.476290 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d333e23-8c2f-44af-b5de-4ae19d6263e5" path="/var/lib/kubelet/pods/9d333e23-8c2f-44af-b5de-4ae19d6263e5/volumes" Oct 31 21:55:23.108622 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.108576 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr"] Oct 31 21:55:23.109622 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.109598 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9d333e23-8c2f-44af-b5de-4ae19d6263e5" containerName="main" Oct 31 21:55:23.109705 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.109626 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d333e23-8c2f-44af-b5de-4ae19d6263e5" containerName="main" Oct 31 21:55:23.109825 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.109813 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d333e23-8c2f-44af-b5de-4ae19d6263e5" containerName="main" Oct 31 21:55:23.115980 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.115948 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.151550 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.151515 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-f1128d-serving-cert\"" Oct 31 21:55:23.156474 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.156445 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr"] Oct 31 21:55:23.212502 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.212449 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-maas-api-tls\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.212700 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.212672 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwqs7\" (UniqueName: \"kubernetes.io/projected/0f002997-8ad7-4865-96b0-16a73efdcaf2-kube-api-access-xwqs7\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.212790 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.212716 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-metrics-certs\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.212912 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.212892 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f002997-8ad7-4865-96b0-16a73efdcaf2-openshift-service-ca\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.313694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.313654 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwqs7\" (UniqueName: \"kubernetes.io/projected/0f002997-8ad7-4865-96b0-16a73efdcaf2-kube-api-access-xwqs7\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.313694 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.313698 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-metrics-certs\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.314011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.313792 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f002997-8ad7-4865-96b0-16a73efdcaf2-openshift-service-ca\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.314011 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.313910 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-maas-api-tls\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.315518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.315477 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f002997-8ad7-4865-96b0-16a73efdcaf2-openshift-service-ca\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.317165 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.317137 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-metrics-certs\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.317895 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.317862 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-maas-api-tls\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.338088 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.338051 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwqs7\" (UniqueName: \"kubernetes.io/projected/0f002997-8ad7-4865-96b0-16a73efdcaf2-kube-api-access-xwqs7\") pod \"maas-api-e2e-autores-f1128d-775cb56889-qs8pr\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.437213 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.437167 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:23.767016 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:23.766984 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr"] Oct 31 21:55:23.768371 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:55:23.768317 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f002997_8ad7_4865_96b0_16a73efdcaf2.slice/crio-7c2848d670353189b1ae375236c2e18272ed879150a42bb33d6431a43c2e108c WatchSource:0}: Error finding container 7c2848d670353189b1ae375236c2e18272ed879150a42bb33d6431a43c2e108c: Status 404 returned error can't find the container with id 7c2848d670353189b1ae375236c2e18272ed879150a42bb33d6431a43c2e108c Oct 31 21:55:24.488356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:24.488178 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" event={"ID":"0f002997-8ad7-4865-96b0-16a73efdcaf2","Type":"ContainerStarted","Data":"81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b"} Oct 31 21:55:24.488356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:24.488239 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" event={"ID":"0f002997-8ad7-4865-96b0-16a73efdcaf2","Type":"ContainerStarted","Data":"7c2848d670353189b1ae375236c2e18272ed879150a42bb33d6431a43c2e108c"} Oct 31 21:55:24.488356 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:24.488286 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:24.530968 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:24.527956 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" podStartSLOduration=1.115765705 podStartE2EDuration="1.527927698s" totalImagesPullingTime="412.161993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:55:24.525611643 +0000 UTC m=+2371.218569032" watchObservedRunningTime="2026-10-31 21:55:24.527927698 +0000 UTC m=+2371.220885092" Oct 31 21:55:30.501696 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:30.501651 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:55:32.543792 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:55:32.543748 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:55:44.400306 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.400263 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27"] Oct 31 21:55:44.406437 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.406409 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.430864 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.430833 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-fac043-serving-cert\"" Oct 31 21:55:44.437610 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.437575 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27"] Oct 31 21:55:44.528779 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.528738 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqc4\" (UniqueName: \"kubernetes.io/projected/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-kube-api-access-jpqc4\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.528981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.528867 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-metrics-certs\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.528981 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.528905 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.529060 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.529021 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.632100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.630078 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqc4\" (UniqueName: \"kubernetes.io/projected/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-kube-api-access-jpqc4\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.632100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.630163 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-metrics-certs\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.632100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.630212 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.632100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.630304 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.632100 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.631971 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.635889 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.635418 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.635889 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.635446 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-metrics-certs\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.653274 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.653205 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqc4\" (UniqueName: \"kubernetes.io/projected/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-kube-api-access-jpqc4\") pod \"maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:44.722549 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:44.722505 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:45.036414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:45.036382 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27"] Oct 31 21:55:45.037573 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:55:45.037537 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb5b051_50c9_4a83_a6d5_d49c87a93b07.slice/crio-66563a692fbb73c162b52e733fe21de166d387be5399a942b9882fe089cad781 WatchSource:0}: Error finding container 66563a692fbb73c162b52e733fe21de166d387be5399a942b9882fe089cad781: Status 404 returned error can't find the container with id 66563a692fbb73c162b52e733fe21de166d387be5399a942b9882fe089cad781 Oct 31 21:55:45.670749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:45.670642 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" event={"ID":"8cb5b051-50c9-4a83-a6d5-d49c87a93b07","Type":"ContainerStarted","Data":"1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede"} Oct 31 21:55:45.670749 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:45.670710 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" event={"ID":"8cb5b051-50c9-4a83-a6d5-d49c87a93b07","Type":"ContainerStarted","Data":"66563a692fbb73c162b52e733fe21de166d387be5399a942b9882fe089cad781"} Oct 31 21:55:45.671230 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:45.670751 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:45.714661 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:45.714585 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" podStartSLOduration=1.343689542 podStartE2EDuration="1.714562555s" totalImagesPullingTime="370.873013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:55:45.708638982 +0000 UTC m=+2392.401596377" watchObservedRunningTime="2026-10-31 21:55:45.714562555 +0000 UTC m=+2392.407519948" Oct 31 21:55:51.685536 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:51.685485 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 21:55:53.954216 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:53.954185 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:55:53.961199 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:55:53.961172 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 21:56:02.540639 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:56:02.540588 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:56:04.789299 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.789262 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh"] Oct 31 21:56:04.795651 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.795621 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.821354 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.821302 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-4a73b1-serving-cert\"" Oct 31 21:56:04.823652 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.822860 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh"] Oct 31 21:56:04.830566 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.830522 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a248b6d-2243-4d70-92c4-73dec22d0d7d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.830878 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.830847 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.831134 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.831029 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-metrics-certs\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.831134 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.831094 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfncs\" (UniqueName: \"kubernetes.io/projected/3a248b6d-2243-4d70-92c4-73dec22d0d7d-kube-api-access-vfncs\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.932031 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.931989 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a248b6d-2243-4d70-92c4-73dec22d0d7d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.932262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.932061 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.932262 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.932215 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-metrics-certs\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.932262 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:56:04.932244 5188 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-serving-cert: secret "maas-api-e2e-extmod-b-4a73b1-serving-cert" not found Oct 31 21:56:04.932471 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.932268 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfncs\" (UniqueName: \"kubernetes.io/projected/3a248b6d-2243-4d70-92c4-73dec22d0d7d-kube-api-access-vfncs\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.932471 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:56:04.932347 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls podName:3a248b6d-2243-4d70-92c4-73dec22d0d7d nodeName:}" failed. No retries permitted until 2026-10-31 21:56:05.432312214 +0000 UTC m=+2412.125269599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls") pod "maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" (UID: "3a248b6d-2243-4d70-92c4-73dec22d0d7d") : secret "maas-api-e2e-extmod-b-4a73b1-serving-cert" not found Oct 31 21:56:04.932872 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.932848 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a248b6d-2243-4d70-92c4-73dec22d0d7d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.934946 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.934917 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-metrics-certs\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:04.953918 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:04.953882 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfncs\" (UniqueName: \"kubernetes.io/projected/3a248b6d-2243-4d70-92c4-73dec22d0d7d-kube-api-access-vfncs\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:05.438067 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:05.438017 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:05.440654 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:05.440616 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:05.711421 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:05.711267 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:06.067484 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:06.067454 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh"] Oct 31 21:56:06.851059 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:06.851011 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" event={"ID":"3a248b6d-2243-4d70-92c4-73dec22d0d7d","Type":"ContainerStarted","Data":"b73a520d08f1eb87af86ad7a09d468062e996f2bd7fbd6bef82eede0afc046f1"} Oct 31 21:56:06.851290 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:06.851268 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:06.851384 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:06.851317 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" event={"ID":"3a248b6d-2243-4d70-92c4-73dec22d0d7d","Type":"ContainerStarted","Data":"89731e9f700328a2f21b1bee99f907cd6287ec7d2ce8e0630f118f89af35c3d0"} Oct 31 21:56:06.895221 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:06.895154 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" podStartSLOduration=2.631154508 podStartE2EDuration="2.895138542s" totalImagesPullingTime="263.984034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:56:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:56:06.891822933 +0000 UTC m=+2413.584780326" watchObservedRunningTime="2026-10-31 21:56:06.895138542 +0000 UTC m=+2413.588095933" Oct 31 21:56:12.864479 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:12.864445 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 21:56:32.540769 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:56:32.540723 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:56:48.603496 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:48.603452 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94"] Oct 31 21:56:48.604586 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:48.604547 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" containerID="cri-o://406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907" gracePeriod=30 Oct 31 21:56:53.945286 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.945255 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:56:53.977518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.976420 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-socket\") pod \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " Oct 31 21:56:53.977518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.976539 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-data\") pod \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " Oct 31 21:56:53.977518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.976566 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-envoy\") pod \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " Oct 31 21:56:53.977518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.976614 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-certs\") pod \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " Oct 31 21:56:53.977518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.976660 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istiod-ca-cert\") pod \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " Oct 31 21:56:53.977518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.976698 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-podinfo\") pod \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " Oct 31 21:56:53.977518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.976724 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-token\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-token\") pod \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " Oct 31 21:56:53.977518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.976773 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-kube-api-access-jdkb7\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-kube-api-access-jdkb7\") pod \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " Oct 31 21:56:53.977518 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.976848 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-credential-socket\") pod \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\" (UID: \"ec53a0f8-ecca-4451-8e0d-deb15db413c5\") " Oct 31 21:56:53.978191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.977678 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-certs" pod "ec53a0f8-ecca-4451-8e0d-deb15db413c5" (UID: "ec53a0f8-ecca-4451-8e0d-deb15db413c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:56:53.978191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.977899 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-socket" pod "ec53a0f8-ecca-4451-8e0d-deb15db413c5" (UID: "ec53a0f8-ecca-4451-8e0d-deb15db413c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:56:53.978191 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.978085 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-data" pod "ec53a0f8-ecca-4451-8e0d-deb15db413c5" (UID: "ec53a0f8-ecca-4451-8e0d-deb15db413c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:56:53.978671 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.978460 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-credential-socket" pod "ec53a0f8-ecca-4451-8e0d-deb15db413c5" (UID: "ec53a0f8-ecca-4451-8e0d-deb15db413c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:56:53.980915 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.980594 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istiod-ca-cert" pod "ec53a0f8-ecca-4451-8e0d-deb15db413c5" (UID: "ec53a0f8-ecca-4451-8e0d-deb15db413c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:56:53.985464 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:53.984277 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-kube-api-access-jdkb7" pod "ec53a0f8-ecca-4451-8e0d-deb15db413c5" (UID: "ec53a0f8-ecca-4451-8e0d-deb15db413c5"). InnerVolumeSpecName "kube-api-access-jdkb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:56:54.012557 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.009150 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-envoy" pod "ec53a0f8-ecca-4451-8e0d-deb15db413c5" (UID: "ec53a0f8-ecca-4451-8e0d-deb15db413c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:56:54.015870 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.015206 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-podinfo" pod "ec53a0f8-ecca-4451-8e0d-deb15db413c5" (UID: "ec53a0f8-ecca-4451-8e0d-deb15db413c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:56:54.042390 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.040533 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-token" pod "ec53a0f8-ecca-4451-8e0d-deb15db413c5" (UID: "ec53a0f8-ecca-4451-8e0d-deb15db413c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:56:54.082348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.078467 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:56:54.082348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.078513 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:56:54.082348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.078526 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:56:54.082348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.078544 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:56:54.082348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.078567 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdkb7\" (UniqueName: \"kubernetes.io/projected/ec53a0f8-ecca-4451-8e0d-deb15db413c5-kube-api-access-jdkb7\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:56:54.082348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.078584 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:56:54.082348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.078598 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:56:54.082348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.078618 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:56:54.082348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.078635 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec53a0f8-ecca-4451-8e0d-deb15db413c5-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:56:54.241056 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.240972 5188 generic.go:356] "Generic (PLEG): container finished" podID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerID="406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907" exitCode=0 Oct 31 21:56:54.241233 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.241080 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" Oct 31 21:56:54.241233 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.241092 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" event={"ID":"ec53a0f8-ecca-4451-8e0d-deb15db413c5","Type":"ContainerDied","Data":"406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907"} Oct 31 21:56:54.241233 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.241160 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94" event={"ID":"ec53a0f8-ecca-4451-8e0d-deb15db413c5","Type":"ContainerDied","Data":"01cb2fcd2f6b802f51ecd05c180b79ca4d63041b660d9a26b3a29c07d79a0b51"} Oct 31 21:56:54.241233 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.241182 5188 scope.go:122] "RemoveContainer" containerID="406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907" Oct 31 21:56:54.252752 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.252723 5188 scope.go:122] "RemoveContainer" containerID="406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907" Oct 31 21:56:54.253049 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:56:54.253024 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907\": container with ID starting with 406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907 not found: ID does not exist" containerID="406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907" Oct 31 21:56:54.253127 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.253062 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907"} err="failed to get container status \"406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907\": rpc error: code = NotFound desc = could not find container \"406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907\": container with ID starting with 406ff54c63e762891d3070d8c6bfcc66864f87eda13b5cb675b4460d1db66907 not found: ID does not exist" Oct 31 21:56:54.287766 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.287735 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94"] Oct 31 21:56:54.291793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:54.291769 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-xjb94"] Oct 31 21:56:55.477089 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:56:55.477055 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" path="/var/lib/kubelet/pods/ec53a0f8-ecca-4451-8e0d-deb15db413c5/volumes" Oct 31 21:57:02.539109 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:57:02.539068 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:57:10.010099 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.009985 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9"] Oct 31 21:57:10.010963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.010940 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" Oct 31 21:57:10.010963 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.010965 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" Oct 31 21:57:10.011139 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.011127 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec53a0f8-ecca-4451-8e0d-deb15db413c5" containerName="istio-proxy" Oct 31 21:57:10.016572 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.016546 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.035254 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.035225 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2dbae9-serving-cert\"" Oct 31 21:57:10.039848 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.039820 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9"] Oct 31 21:57:10.104987 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.104934 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.105192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.105117 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-metrics-certs\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.105273 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.105251 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rtfs\" (UniqueName: \"kubernetes.io/projected/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-kube-api-access-4rtfs\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.105363 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.105304 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.206300 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.206259 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-metrics-certs\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.206541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.206436 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rtfs\" (UniqueName: \"kubernetes.io/projected/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-kube-api-access-4rtfs\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.206541 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.206496 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.206675 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.206578 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.207276 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.207249 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.209662 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.209637 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.209796 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.209681 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-metrics-certs\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.235889 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.235414 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rtfs\" (UniqueName: \"kubernetes.io/projected/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-kube-api-access-4rtfs\") pod \"maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.333259 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.333157 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:10.647691 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:10.647657 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9"] Oct 31 21:57:10.648789 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:57:10.648755 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10512159_6a38_4a9d_b7d0_fd3f7aaace1c.slice/crio-fad76cea3383f304761e3e951e14ab2733f33530279cadc8efc1e68ee108fd17 WatchSource:0}: Error finding container fad76cea3383f304761e3e951e14ab2733f33530279cadc8efc1e68ee108fd17: Status 404 returned error can't find the container with id fad76cea3383f304761e3e951e14ab2733f33530279cadc8efc1e68ee108fd17 Oct 31 21:57:11.420798 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:11.420749 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" event={"ID":"10512159-6a38-4a9d-b7d0-fd3f7aaace1c","Type":"ContainerStarted","Data":"f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4"} Oct 31 21:57:11.421385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:11.420819 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" event={"ID":"10512159-6a38-4a9d-b7d0-fd3f7aaace1c","Type":"ContainerStarted","Data":"fad76cea3383f304761e3e951e14ab2733f33530279cadc8efc1e68ee108fd17"} Oct 31 21:57:11.421385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:11.420886 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:11.462148 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:11.462081 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" podStartSLOduration=2.113756902 podStartE2EDuration="2.462062372s" totalImagesPullingTime="348.30547ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:57:11.458254589 +0000 UTC m=+2478.151211982" watchObservedRunningTime="2026-10-31 21:57:11.462062372 +0000 UTC m=+2478.155019765" Oct 31 21:57:17.434868 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:17.434832 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 21:57:18.909237 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:18.909196 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr"] Oct 31 21:57:18.909804 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:18.909490 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" podUID="0f002997-8ad7-4865-96b0-16a73efdcaf2" containerName="maas-api" containerID="cri-o://81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b" gracePeriod=30 Oct 31 21:57:19.247911 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.247883 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:57:19.351844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.351794 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-metrics-certs\") pod \"0f002997-8ad7-4865-96b0-16a73efdcaf2\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " Oct 31 21:57:19.352065 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.351971 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f002997-8ad7-4865-96b0-16a73efdcaf2-kube-api-access-xwqs7\" (UniqueName: \"kubernetes.io/projected/0f002997-8ad7-4865-96b0-16a73efdcaf2-kube-api-access-xwqs7\") pod \"0f002997-8ad7-4865-96b0-16a73efdcaf2\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " Oct 31 21:57:19.352065 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.352006 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f002997-8ad7-4865-96b0-16a73efdcaf2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f002997-8ad7-4865-96b0-16a73efdcaf2-openshift-service-ca\") pod \"0f002997-8ad7-4865-96b0-16a73efdcaf2\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " Oct 31 21:57:19.352065 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.352047 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-maas-api-tls\") pod \"0f002997-8ad7-4865-96b0-16a73efdcaf2\" (UID: \"0f002997-8ad7-4865-96b0-16a73efdcaf2\") " Oct 31 21:57:19.354066 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.353983 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f002997-8ad7-4865-96b0-16a73efdcaf2-openshift-service-ca" pod "0f002997-8ad7-4865-96b0-16a73efdcaf2" (UID: "0f002997-8ad7-4865-96b0-16a73efdcaf2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:57:19.356728 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.356640 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-metrics-certs" pod "0f002997-8ad7-4865-96b0-16a73efdcaf2" (UID: "0f002997-8ad7-4865-96b0-16a73efdcaf2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:57:19.361578 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.361540 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f002997-8ad7-4865-96b0-16a73efdcaf2-kube-api-access-xwqs7" pod "0f002997-8ad7-4865-96b0-16a73efdcaf2" (UID: "0f002997-8ad7-4865-96b0-16a73efdcaf2"). InnerVolumeSpecName "kube-api-access-xwqs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:57:19.363238 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.363131 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-maas-api-tls" pod "0f002997-8ad7-4865-96b0-16a73efdcaf2" (UID: "0f002997-8ad7-4865-96b0-16a73efdcaf2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:57:19.455348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.454470 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwqs7\" (UniqueName: \"kubernetes.io/projected/0f002997-8ad7-4865-96b0-16a73efdcaf2-kube-api-access-xwqs7\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:57:19.455348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.454516 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f002997-8ad7-4865-96b0-16a73efdcaf2-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:57:19.455348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.454532 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:57:19.455348 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.454545 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f002997-8ad7-4865-96b0-16a73efdcaf2-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 21:57:19.500608 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.500564 5188 generic.go:356] "Generic (PLEG): container finished" podID="0f002997-8ad7-4865-96b0-16a73efdcaf2" containerID="81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b" exitCode=0 Oct 31 21:57:19.500816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.500617 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" Oct 31 21:57:19.500816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.500642 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" event={"ID":"0f002997-8ad7-4865-96b0-16a73efdcaf2","Type":"ContainerDied","Data":"81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b"} Oct 31 21:57:19.500816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.500705 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr" event={"ID":"0f002997-8ad7-4865-96b0-16a73efdcaf2","Type":"ContainerDied","Data":"7c2848d670353189b1ae375236c2e18272ed879150a42bb33d6431a43c2e108c"} Oct 31 21:57:19.500816 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.500733 5188 scope.go:122] "RemoveContainer" containerID="81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b" Oct 31 21:57:19.513583 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.513559 5188 scope.go:122] "RemoveContainer" containerID="81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b" Oct 31 21:57:19.513942 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:57:19.513913 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b\": container with ID starting with 81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b not found: ID does not exist" containerID="81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b" Oct 31 21:57:19.514053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.513947 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b"} err="failed to get container status \"81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b\": rpc error: code = NotFound desc = could not find container \"81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b\": container with ID starting with 81f3d74d84cca4e20a714b18ba19b49f15cd4a4d3cabc5ec6815242be1acd55b not found: ID does not exist" Oct 31 21:57:19.547358 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.547302 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr"] Oct 31 21:57:19.552920 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:19.552887 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f1128d-775cb56889-qs8pr"] Oct 31 21:57:21.475053 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:57:21.475010 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f002997-8ad7-4865-96b0-16a73efdcaf2" path="/var/lib/kubelet/pods/0f002997-8ad7-4865-96b0-16a73efdcaf2/volumes" Oct 31 21:57:32.538500 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:57:32.538464 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:58:02.539285 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:58:02.539243 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:58:13.493968 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.493925 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf"] Oct 31 21:58:13.494979 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.494958 5188 cpu_manager.go:414] "containerMap: removing container" podUID="0f002997-8ad7-4865-96b0-16a73efdcaf2" containerName="maas-api" Oct 31 21:58:13.495047 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.494983 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f002997-8ad7-4865-96b0-16a73efdcaf2" containerName="maas-api" Oct 31 21:58:13.495212 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.495201 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f002997-8ad7-4865-96b0-16a73efdcaf2" containerName="maas-api" Oct 31 21:58:13.501025 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.500997 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.517228 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.517196 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf"] Oct 31 21:58:13.522263 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.522231 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-4275b9-serving-cert\"" Oct 31 21:58:13.578222 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.578176 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.578523 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.578492 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da10e874-0602-4ca3-86aa-500bb5b32448-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.578704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.578546 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-metrics-certs\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.578704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.578676 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdd6q\" (UniqueName: \"kubernetes.io/projected/da10e874-0602-4ca3-86aa-500bb5b32448-kube-api-access-xdd6q\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.680631 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.680582 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdd6q\" (UniqueName: \"kubernetes.io/projected/da10e874-0602-4ca3-86aa-500bb5b32448-kube-api-access-xdd6q\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.680861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.680679 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.680861 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.680842 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da10e874-0602-4ca3-86aa-500bb5b32448-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.680992 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.680875 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-metrics-certs\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.681793 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.681754 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da10e874-0602-4ca3-86aa-500bb5b32448-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.683835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.683804 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-metrics-certs\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.683985 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.683804 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-maas-api-tls\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.702133 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.702097 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdd6q\" (UniqueName: \"kubernetes.io/projected/da10e874-0602-4ca3-86aa-500bb5b32448-kube-api-access-xdd6q\") pod \"maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:13.819746 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:13.819631 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:14.140173 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:14.140143 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf"] Oct 31 21:58:14.140984 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:58:14.140955 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda10e874_0602_4ca3_86aa_500bb5b32448.slice/crio-941678448ee289a5a9b804c9b295addc0f7da3a4525328cb674691205f5e69ed WatchSource:0}: Error finding container 941678448ee289a5a9b804c9b295addc0f7da3a4525328cb674691205f5e69ed: Status 404 returned error can't find the container with id 941678448ee289a5a9b804c9b295addc0f7da3a4525328cb674691205f5e69ed Oct 31 21:58:14.142458 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:14.142438 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:58:14.978878 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:14.978833 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" event={"ID":"da10e874-0602-4ca3-86aa-500bb5b32448","Type":"ContainerStarted","Data":"24491d40dc0d767cfa5e32d8e8b738d439858dea3949dff2cb4008e6c3db85fc"} Oct 31 21:58:14.979466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:14.978893 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" event={"ID":"da10e874-0602-4ca3-86aa-500bb5b32448","Type":"ContainerStarted","Data":"941678448ee289a5a9b804c9b295addc0f7da3a4525328cb674691205f5e69ed"} Oct 31 21:58:14.979466 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:14.978970 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:15.020591 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:15.020513 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" podStartSLOduration=1.650649528 podStartE2EDuration="2.020491321s" totalImagesPullingTime="369.841793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:58:15.016138451 +0000 UTC m=+2541.709095844" watchObservedRunningTime="2026-10-31 21:58:15.020491321 +0000 UTC m=+2541.713448719" Oct 31 21:58:20.993260 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:58:20.993224 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 21:58:32.539731 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:58:32.539690 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:59:02.539921 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:59:02.539879 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:59:07.267704 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.267627 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw"] Oct 31 21:59:07.287385 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.284777 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:07.291346 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.289234 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw"] Oct 31 21:59:07.306414 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.305339 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a5affe\"" Oct 31 21:59:07.312429 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.310540 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a5affe-dockercfg-82rzx\"" Oct 31 21:59:07.452074 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.452028 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpgh\" (UniqueName: \"kubernetes.io/projected/95a84925-746c-4331-8f5a-197e34cd274d-kube-api-access-4vpgh\") pod \"payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw\" (UID: \"95a84925-746c-4331-8f5a-197e34cd274d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:07.452267 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.452224 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a84925-746c-4331-8f5a-197e34cd274d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw\" (UID: \"95a84925-746c-4331-8f5a-197e34cd274d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:07.554018 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.553933 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a84925-746c-4331-8f5a-197e34cd274d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw\" (UID: \"95a84925-746c-4331-8f5a-197e34cd274d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:07.554192 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.554070 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpgh\" (UniqueName: \"kubernetes.io/projected/95a84925-746c-4331-8f5a-197e34cd274d-kube-api-access-4vpgh\") pod \"payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw\" (UID: \"95a84925-746c-4331-8f5a-197e34cd274d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:07.554738 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.554706 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a84925-746c-4331-8f5a-197e34cd274d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw\" (UID: \"95a84925-746c-4331-8f5a-197e34cd274d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:07.575900 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.575856 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpgh\" (UniqueName: \"kubernetes.io/projected/95a84925-746c-4331-8f5a-197e34cd274d-kube-api-access-4vpgh\") pod \"payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw\" (UID: \"95a84925-746c-4331-8f5a-197e34cd274d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:07.648820 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.648781 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:07.973567 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:07.973534 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw"] Oct 31 21:59:07.973835 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:59:07.973810 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a84925_746c_4331_8f5a_197e34cd274d.slice/crio-630d387fb928da3ab68622b39963a18882d0ffcd4fd6dbcd47aa322952ecf6c9 WatchSource:0}: Error finding container 630d387fb928da3ab68622b39963a18882d0ffcd4fd6dbcd47aa322952ecf6c9: Status 404 returned error can't find the container with id 630d387fb928da3ab68622b39963a18882d0ffcd4fd6dbcd47aa322952ecf6c9 Oct 31 21:59:08.433697 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:08.433655 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" event={"ID":"95a84925-746c-4331-8f5a-197e34cd274d","Type":"ContainerStarted","Data":"fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914"} Oct 31 21:59:08.434220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:08.433720 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" event={"ID":"95a84925-746c-4331-8f5a-197e34cd274d","Type":"ContainerStarted","Data":"630d387fb928da3ab68622b39963a18882d0ffcd4fd6dbcd47aa322952ecf6c9"} Oct 31 21:59:08.434220 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:08.433747 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:08.472373 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:08.472276 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" podStartSLOduration=1.472254255 podStartE2EDuration="1.472254255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:59:08.469653397 +0000 UTC m=+2595.162610790" watchObservedRunningTime="2026-10-31 21:59:08.472254255 +0000 UTC m=+2595.165211648" Oct 31 21:59:19.446064 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:19.446020 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 21:59:32.540793 ip-10-0-131-21 kubenswrapper[5188]: E1031 21:59:32.540756 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 21:59:44.306308 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.306266 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn"] Oct 31 21:59:44.311955 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.311933 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.329508 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.329474 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn"] Oct 31 21:59:44.352369 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.333563 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a5affe-serving-cert\"" Oct 31 21:59:44.399615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.399583 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-metrics-certs\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.399615 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.399620 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.399835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.399708 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e142d5-6162-47b4-9f68-c1c16aa8a778-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.399835 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.399809 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc924\" (UniqueName: \"kubernetes.io/projected/83e142d5-6162-47b4-9f68-c1c16aa8a778-kube-api-access-nc924\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.501003 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.500964 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-metrics-certs\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.501003 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.501010 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.501246 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.501057 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e142d5-6162-47b4-9f68-c1c16aa8a778-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.501246 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.501099 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc924\" (UniqueName: \"kubernetes.io/projected/83e142d5-6162-47b4-9f68-c1c16aa8a778-kube-api-access-nc924\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.501876 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.501849 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e142d5-6162-47b4-9f68-c1c16aa8a778-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.503711 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.503680 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.503844 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.503730 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-metrics-certs\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.546664 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.546586 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc924\" (UniqueName: \"kubernetes.io/projected/83e142d5-6162-47b4-9f68-c1c16aa8a778-kube-api-access-nc924\") pod \"maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.630919 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.630878 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:44.939932 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:44.939891 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn"] Oct 31 21:59:44.941452 ip-10-0-131-21 kubenswrapper[5188]: W1031 21:59:44.941406 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e142d5_6162_47b4_9f68_c1c16aa8a778.slice/crio-269e520eca3c9647c5037331f65bda5580ad48fa93a9b263466e033f0a708167 WatchSource:0}: Error finding container 269e520eca3c9647c5037331f65bda5580ad48fa93a9b263466e033f0a708167: Status 404 returned error can't find the container with id 269e520eca3c9647c5037331f65bda5580ad48fa93a9b263466e033f0a708167 Oct 31 21:59:45.757965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:45.757868 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" event={"ID":"83e142d5-6162-47b4-9f68-c1c16aa8a778","Type":"ContainerStarted","Data":"ae2ecb72d4fbd2392d93d1304b7a0447070e606b7804c28323a76845f4b79f1e"} Oct 31 21:59:45.757965 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:45.757935 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" event={"ID":"83e142d5-6162-47b4-9f68-c1c16aa8a778","Type":"ContainerStarted","Data":"269e520eca3c9647c5037331f65bda5580ad48fa93a9b263466e033f0a708167"} Oct 31 21:59:45.758454 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:45.758148 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 21:59:45.803607 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:45.803541 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" podStartSLOduration=1.363485723 podStartE2EDuration="1.803523613s" totalImagesPullingTime="440.03789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:59:45.799312774 +0000 UTC m=+2632.492270168" watchObservedRunningTime="2026-10-31 21:59:45.803523613 +0000 UTC m=+2632.496481007" Oct 31 21:59:51.771198 ip-10-0-131-21 kubenswrapper[5188]: I1031 21:59:51.771163 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 22:00:00.245913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.245864 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5"] Oct 31 22:00:00.251370 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.251317 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:00.267555 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.267246 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5"] Oct 31 22:00:00.275121 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.275090 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:00:00.336515 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.336471 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jqz9\" (UniqueName: \"kubernetes.io/projected/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-kube-api-access-5jqz9\") pod \"maas-api-key-cleanup-29891400-sccq5\" (UID: \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:00.336761 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.336741 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891400-sccq5\" (UID: \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:00.438544 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.438510 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jqz9\" (UniqueName: \"kubernetes.io/projected/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-kube-api-access-5jqz9\") pod \"maas-api-key-cleanup-29891400-sccq5\" (UID: \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:00.438799 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.438682 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891400-sccq5\" (UID: \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:00.439522 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.439497 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891400-sccq5\" (UID: \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:00.459968 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.459928 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jqz9\" (UniqueName: \"kubernetes.io/projected/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-kube-api-access-5jqz9\") pod \"maas-api-key-cleanup-29891400-sccq5\" (UID: \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:00.585055 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.584969 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:00.907445 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:00.907236 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5"] Oct 31 22:00:00.907445 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:00:00.907392 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c9e2b1_0362_4dae_9fd2_ab1587abd513.slice/crio-93c3fb3891b983b7260d19eab02c646cd0e1b775fe4120ab13e95b6ac82a9f01 WatchSource:0}: Error finding container 93c3fb3891b983b7260d19eab02c646cd0e1b775fe4120ab13e95b6ac82a9f01: Status 404 returned error can't find the container with id 93c3fb3891b983b7260d19eab02c646cd0e1b775fe4120ab13e95b6ac82a9f01 Oct 31 22:00:01.901782 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:01.901732 5188 generic.go:356] "Generic (PLEG): container finished" podID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerID="e3ed03dc80e0c6b69c11aaa38156488995b094e55df445894948dd71779bd215" exitCode=60 Oct 31 22:00:01.902251 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:01.901821 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" event={"ID":"d8c9e2b1-0362-4dae-9fd2-ab1587abd513","Type":"ContainerDied","Data":"e3ed03dc80e0c6b69c11aaa38156488995b094e55df445894948dd71779bd215"} Oct 31 22:00:01.902251 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:01.901896 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" event={"ID":"d8c9e2b1-0362-4dae-9fd2-ab1587abd513","Type":"ContainerStarted","Data":"93c3fb3891b983b7260d19eab02c646cd0e1b775fe4120ab13e95b6ac82a9f01"} Oct 31 22:00:01.902405 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:01.902288 5188 scope.go:122] "RemoveContainer" containerID="e3ed03dc80e0c6b69c11aaa38156488995b094e55df445894948dd71779bd215" Oct 31 22:00:02.539012 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:00:02.538968 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:00:02.913618 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:02.913581 5188 generic.go:356] "Generic (PLEG): container finished" podID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerID="05c7210b4755d56c742b07a1abeda47429aaf325d8442590873422e996e09a48" exitCode=60 Oct 31 22:00:02.914113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:02.913667 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" event={"ID":"d8c9e2b1-0362-4dae-9fd2-ab1587abd513","Type":"ContainerDied","Data":"05c7210b4755d56c742b07a1abeda47429aaf325d8442590873422e996e09a48"} Oct 31 22:00:02.914113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:02.913741 5188 scope.go:122] "RemoveContainer" containerID="e3ed03dc80e0c6b69c11aaa38156488995b094e55df445894948dd71779bd215" Oct 31 22:00:02.914113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:02.914098 5188 scope.go:122] "RemoveContainer" containerID="05c7210b4755d56c742b07a1abeda47429aaf325d8442590873422e996e09a48" Oct 31 22:00:02.914532 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:00:02.914503 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891400-sccq5_odh-ai-gateway-infra(d8c9e2b1-0362-4dae-9fd2-ab1587abd513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" Oct 31 22:00:03.926825 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:03.926752 5188 scope.go:122] "RemoveContainer" containerID="05c7210b4755d56c742b07a1abeda47429aaf325d8442590873422e996e09a48" Oct 31 22:00:03.927355 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:00:03.927096 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891400-sccq5_odh-ai-gateway-infra(d8c9e2b1-0362-4dae-9fd2-ab1587abd513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" Oct 31 22:00:16.468238 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:16.468199 5188 scope.go:122] "RemoveContainer" containerID="05c7210b4755d56c742b07a1abeda47429aaf325d8442590873422e996e09a48" Oct 31 22:00:17.049010 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:17.048965 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" event={"ID":"d8c9e2b1-0362-4dae-9fd2-ab1587abd513","Type":"ContainerStarted","Data":"78e15ed1599d89d30d792259442f17596d90cd9684c4ccdf4795f91fe487598d"} Oct 31 22:00:17.085692 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:17.085631 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" podStartSLOduration=17.085613205 podStartE2EDuration="17.085613205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:17.084221848 +0000 UTC m=+2663.777179241" watchObservedRunningTime="2026-10-31 22:00:17.085613205 +0000 UTC m=+2663.778570599" Oct 31 22:00:18.061632 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:18.061586 5188 generic.go:356] "Generic (PLEG): container finished" podID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerID="78e15ed1599d89d30d792259442f17596d90cd9684c4ccdf4795f91fe487598d" exitCode=60 Oct 31 22:00:18.062173 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:18.061626 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" event={"ID":"d8c9e2b1-0362-4dae-9fd2-ab1587abd513","Type":"ContainerDied","Data":"78e15ed1599d89d30d792259442f17596d90cd9684c4ccdf4795f91fe487598d"} Oct 31 22:00:18.062173 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:18.061732 5188 scope.go:122] "RemoveContainer" containerID="05c7210b4755d56c742b07a1abeda47429aaf325d8442590873422e996e09a48" Oct 31 22:00:18.062313 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:18.062189 5188 scope.go:122] "RemoveContainer" containerID="78e15ed1599d89d30d792259442f17596d90cd9684c4ccdf4795f91fe487598d" Oct 31 22:00:18.062660 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:00:18.062633 5188 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891400-sccq5_odh-ai-gateway-infra(d8c9e2b1-0362-4dae-9fd2-ab1587abd513)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" Oct 31 22:00:18.097231 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:18.097189 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5"] Oct 31 22:00:19.251382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:19.251353 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:19.429871 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:19.429835 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-maas-api-service-ca\") pod \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\" (UID: \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\") " Oct 31 22:00:19.430118 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:19.429958 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-kube-api-access-5jqz9\" (UniqueName: \"kubernetes.io/projected/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-kube-api-access-5jqz9\") pod \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\" (UID: \"d8c9e2b1-0362-4dae-9fd2-ab1587abd513\") " Oct 31 22:00:19.430257 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:19.430231 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-maas-api-service-ca" pod "d8c9e2b1-0362-4dae-9fd2-ab1587abd513" (UID: "d8c9e2b1-0362-4dae-9fd2-ab1587abd513"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:19.430496 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:19.430478 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-maas-api-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.433044 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:19.433006 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-kube-api-access-5jqz9" pod "d8c9e2b1-0362-4dae-9fd2-ab1587abd513" (UID: "d8c9e2b1-0362-4dae-9fd2-ab1587abd513"). InnerVolumeSpecName "kube-api-access-5jqz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:19.531814 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:19.531776 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jqz9\" (UniqueName: \"kubernetes.io/projected/d8c9e2b1-0362-4dae-9fd2-ab1587abd513-kube-api-access-5jqz9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:20.080318 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:20.080279 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" Oct 31 22:00:20.080318 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:20.080309 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5" event={"ID":"d8c9e2b1-0362-4dae-9fd2-ab1587abd513","Type":"ContainerDied","Data":"93c3fb3891b983b7260d19eab02c646cd0e1b775fe4120ab13e95b6ac82a9f01"} Oct 31 22:00:20.080616 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:20.080397 5188 scope.go:122] "RemoveContainer" containerID="78e15ed1599d89d30d792259442f17596d90cd9684c4ccdf4795f91fe487598d" Oct 31 22:00:20.131168 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:20.131128 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5"] Oct 31 22:00:20.141652 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:20.141622 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891400-sccq5"] Oct 31 22:00:21.478798 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:21.478761 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" path="/var/lib/kubelet/pods/d8c9e2b1-0362-4dae-9fd2-ab1587abd513/volumes" Oct 31 22:00:32.546632 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:00:32.546577 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:00:33.808495 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:33.808448 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp"] Oct 31 22:00:33.809040 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:33.808757 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" containerID="cri-o://f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975" gracePeriod=30 Oct 31 22:00:39.113454 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.113415 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 22:00:39.237714 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.237680 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff66395-e99b-411d-a3c5-f397a62f5004-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff66395-e99b-411d-a3c5-f397a62f5004-istiod-ca-cert\") pod \"0ff66395-e99b-411d-a3c5-f397a62f5004\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " Oct 31 22:00:39.237935 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.237772 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-token\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-token\") pod \"0ff66395-e99b-411d-a3c5-f397a62f5004\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " Oct 31 22:00:39.237935 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.237824 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-kube-api-access-6sn2l\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-kube-api-access-6sn2l\") pod \"0ff66395-e99b-411d-a3c5-f397a62f5004\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " Oct 31 22:00:39.237935 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.237882 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-podinfo\") pod \"0ff66395-e99b-411d-a3c5-f397a62f5004\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " Oct 31 22:00:39.237935 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.237929 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-credential-socket\") pod \"0ff66395-e99b-411d-a3c5-f397a62f5004\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " Oct 31 22:00:39.238163 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.238004 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-data\") pod \"0ff66395-e99b-411d-a3c5-f397a62f5004\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " Oct 31 22:00:39.238163 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.238050 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-envoy\") pod \"0ff66395-e99b-411d-a3c5-f397a62f5004\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " Oct 31 22:00:39.238163 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.238121 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-certs\") pod \"0ff66395-e99b-411d-a3c5-f397a62f5004\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " Oct 31 22:00:39.238353 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.238163 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff66395-e99b-411d-a3c5-f397a62f5004-istiod-ca-cert" pod "0ff66395-e99b-411d-a3c5-f397a62f5004" (UID: "0ff66395-e99b-411d-a3c5-f397a62f5004"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:39.238353 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.238259 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-socket\") pod \"0ff66395-e99b-411d-a3c5-f397a62f5004\" (UID: \"0ff66395-e99b-411d-a3c5-f397a62f5004\") " Oct 31 22:00:39.238482 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.238453 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-certs" pod "0ff66395-e99b-411d-a3c5-f397a62f5004" (UID: "0ff66395-e99b-411d-a3c5-f397a62f5004"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.238545 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.238468 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-data" pod "0ff66395-e99b-411d-a3c5-f397a62f5004" (UID: "0ff66395-e99b-411d-a3c5-f397a62f5004"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.238683 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.238661 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-credential-socket" pod "0ff66395-e99b-411d-a3c5-f397a62f5004" (UID: "0ff66395-e99b-411d-a3c5-f397a62f5004"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.238683 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.238674 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-socket" pod "0ff66395-e99b-411d-a3c5-f397a62f5004" (UID: "0ff66395-e99b-411d-a3c5-f397a62f5004"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.239122 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.239099 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.239219 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.239128 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.239219 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.239141 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.239219 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.239155 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff66395-e99b-411d-a3c5-f397a62f5004-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.239219 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.239169 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.240877 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.240846 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-podinfo" pod "0ff66395-e99b-411d-a3c5-f397a62f5004" (UID: "0ff66395-e99b-411d-a3c5-f397a62f5004"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:00:39.240997 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.240898 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-kube-api-access-6sn2l" pod "0ff66395-e99b-411d-a3c5-f397a62f5004" (UID: "0ff66395-e99b-411d-a3c5-f397a62f5004"). InnerVolumeSpecName "kube-api-access-6sn2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:39.240997 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.240899 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-envoy" pod "0ff66395-e99b-411d-a3c5-f397a62f5004" (UID: "0ff66395-e99b-411d-a3c5-f397a62f5004"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.241091 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.241006 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-token" pod "0ff66395-e99b-411d-a3c5-f397a62f5004" (UID: "0ff66395-e99b-411d-a3c5-f397a62f5004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:39.244179 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.244150 5188 generic.go:356] "Generic (PLEG): container finished" podID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerID="f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975" exitCode=0 Oct 31 22:00:39.244389 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.244211 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" Oct 31 22:00:39.244389 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.244215 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" event={"ID":"0ff66395-e99b-411d-a3c5-f397a62f5004","Type":"ContainerDied","Data":"f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975"} Oct 31 22:00:39.244389 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.244275 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp" event={"ID":"0ff66395-e99b-411d-a3c5-f397a62f5004","Type":"ContainerDied","Data":"f22c639aee82852c249ccf0604b8f9b57c48129a7a1d1c32d562f6618847b877"} Oct 31 22:00:39.244389 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.244307 5188 scope.go:122] "RemoveContainer" containerID="f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975" Oct 31 22:00:39.278681 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.278637 5188 scope.go:122] "RemoveContainer" containerID="f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975" Oct 31 22:00:39.279095 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:00:39.279058 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975\": container with ID starting with f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975 not found: ID does not exist" containerID="f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975" Oct 31 22:00:39.279265 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.279108 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975"} err="failed to get container status \"f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975\": rpc error: code = NotFound desc = could not find container \"f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975\": container with ID starting with f05be8ea88f714ebf55e1ead75252086db8fb59c775b40440dff59f7a2a10975 not found: ID does not exist" Oct 31 22:00:39.309546 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.309412 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp"] Oct 31 22:00:39.315493 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.315439 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-hqcwp"] Oct 31 22:00:39.340480 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.340392 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.340480 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.340425 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.340480 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.340435 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sn2l\" (UniqueName: \"kubernetes.io/projected/0ff66395-e99b-411d-a3c5-f397a62f5004-kube-api-access-6sn2l\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.340480 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.340447 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff66395-e99b-411d-a3c5-f397a62f5004-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.474647 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:39.474613 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" path="/var/lib/kubelet/pods/0ff66395-e99b-411d-a3c5-f397a62f5004/volumes" Oct 31 22:00:54.010579 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:54.010548 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:00:54.020886 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:00:54.020857 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:01:02.539145 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:01:02.539104 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:01:32.539155 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:01:32.539109 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:02:02.538788 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:02:02.538743 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:02:32.558014 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:02:32.557968 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:02:41.334477 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.334432 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9"] Oct 31 22:02:41.335469 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335444 5188 cpu_manager.go:414] "containerMap: removing container" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" Oct 31 22:02:41.335469 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335470 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" Oct 31 22:02:41.335655 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335487 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerName="cleanup" Oct 31 22:02:41.335655 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335495 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerName="cleanup" Oct 31 22:02:41.335655 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335520 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerName="cleanup" Oct 31 22:02:41.335655 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335529 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerName="cleanup" Oct 31 22:02:41.335655 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335580 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerName="cleanup" Oct 31 22:02:41.335655 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335588 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerName="cleanup" Oct 31 22:02:41.335918 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335806 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerName="cleanup" Oct 31 22:02:41.335918 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335843 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff66395-e99b-411d-a3c5-f397a62f5004" containerName="istio-proxy" Oct 31 22:02:41.335918 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.335852 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerName="cleanup" Oct 31 22:02:41.340066 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.340044 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.353485 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.353426 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dn2rx\"" Oct 31 22:02:41.373554 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.373518 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9"] Oct 31 22:02:41.480749 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.480700 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.480975 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.480758 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a331fa0-7891-4369-9f99-861558258d5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.480975 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.480882 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6hg4\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-kube-api-access-c6hg4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.480975 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.480934 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.481124 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.481032 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a331fa0-7891-4369-9f99-861558258d5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.481124 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.481053 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.481124 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.481078 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.481252 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.481211 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.481252 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.481241 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.582762 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.582715 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a331fa0-7891-4369-9f99-861558258d5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.582762 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.582760 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583052 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.582786 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583052 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.582843 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583052 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.582949 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583336 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.583294 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583424 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.583365 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583424 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.583400 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583537 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.583409 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583537 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.583475 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a331fa0-7891-4369-9f99-861558258d5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583537 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.583525 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6hg4\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-kube-api-access-c6hg4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583707 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.583564 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583707 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.583580 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.583866 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.583840 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a331fa0-7891-4369-9f99-861558258d5f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.585970 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.585907 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.586137 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.586117 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a331fa0-7891-4369-9f99-861558258d5f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.605251 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.605219 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.605698 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.605362 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6hg4\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-kube-api-access-c6hg4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-vxrb9\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.666572 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.666526 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:41.997380 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:41.997316 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9"] Oct 31 22:02:42.000624 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:42.000582 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 22:02:42.000732 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:42.000676 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 22:02:42.000774 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:42.000729 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 22:02:42.386419 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:42.386372 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" event={"ID":"5a331fa0-7891-4369-9f99-861558258d5f","Type":"ContainerStarted","Data":"0a1605db271b840aa02af7bbaee57d467b7f30469dbebca2d3591123a41d5cf6"} Oct 31 22:02:42.386956 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:42.386433 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" event={"ID":"5a331fa0-7891-4369-9f99-861558258d5f","Type":"ContainerStarted","Data":"0ea69a9a7d5b5333ca59320a957a4e0525ecc7c5df0a6c40bbfff9a74f31d6f8"} Oct 31 22:02:42.427128 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:42.427058 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podStartSLOduration=1.427043213 podStartE2EDuration="1.427043213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:02:42.426571092 +0000 UTC m=+2809.119528486" watchObservedRunningTime="2026-10-31 22:02:42.427043213 +0000 UTC m=+2809.120000605" Oct 31 22:02:42.667189 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:42.667104 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:43.668652 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:43.668610 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vxrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 22:02:43.669115 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:43.668672 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 22:02:44.667287 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:44.667249 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vxrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 22:02:44.667503 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:44.667348 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 22:02:45.666936 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:45.666898 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vxrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 22:02:45.667621 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:45.667586 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 22:02:46.667661 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:46.667426 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vxrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 22:02:46.667661 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:46.667559 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 22:02:47.667305 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:47.667263 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vxrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 22:02:47.667627 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:47.667349 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 22:02:48.666954 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:48.666913 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vxrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 22:02:48.667545 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:48.667011 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 22:02:49.673350 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:49.669492 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vxrb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 22:02:49.673350 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:49.669566 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 22:02:51.091725 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:51.091688 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:51.092257 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:51.092181 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:51.093816 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:51.093794 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:02:56.400685 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.400648 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb"] Oct 31 22:02:56.401636 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.401615 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c9e2b1-0362-4dae-9fd2-ab1587abd513" containerName="cleanup" Oct 31 22:02:56.405524 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.405501 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.428530 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.428499 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb"] Oct 31 22:02:56.430895 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.430865 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.431204 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.431138 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.431555 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.431262 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.431555 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.431305 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.431555 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.431384 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.431555 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.431427 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a019eb-d5f0-41f8-9822-eeb0c265e906-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.431555 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.431490 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rdm\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-kube-api-access-c5rdm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.431555 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.431533 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.431958 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.431578 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.532970 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.532934 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533160 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.532986 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533160 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533034 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533160 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533106 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533160 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533146 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533207 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533248 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a019eb-d5f0-41f8-9822-eeb0c265e906-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533351 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rdm\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-kube-api-access-c5rdm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533404 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533424 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533787 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533493 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.533787 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.533527 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.534139 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.534105 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a019eb-d5f0-41f8-9822-eeb0c265e906-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.534139 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.534134 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.535720 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.535696 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.535922 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.535902 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.554539 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.554507 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.554680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.554664 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rdm\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-kube-api-access-c5rdm\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gjgbb\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:56.730931 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:56.730842 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:57.057544 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:57.057514 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb"] Oct 31 22:02:57.058509 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:02:57.058480 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a019eb_d5f0_41f8_9822_eeb0c265e906.slice/crio-ff73a29a44decb1ae28c756009c61746b2c7cac4cc850890a2c44ede85503f36 WatchSource:0}: Error finding container ff73a29a44decb1ae28c756009c61746b2c7cac4cc850890a2c44ede85503f36: Status 404 returned error can't find the container with id ff73a29a44decb1ae28c756009c61746b2c7cac4cc850890a2c44ede85503f36 Oct 31 22:02:57.061699 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:57.061605 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 22:02:57.061831 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:57.061704 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 22:02:57.061831 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:57.061750 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 22:02:57.541676 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:57.541629 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" event={"ID":"36a019eb-d5f0-41f8-9822-eeb0c265e906","Type":"ContainerStarted","Data":"eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02"} Oct 31 22:02:57.542207 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:57.541704 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" event={"ID":"36a019eb-d5f0-41f8-9822-eeb0c265e906","Type":"ContainerStarted","Data":"ff73a29a44decb1ae28c756009c61746b2c7cac4cc850890a2c44ede85503f36"} Oct 31 22:02:57.587147 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:57.587076 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podStartSLOduration=1.587053802 podStartE2EDuration="1.587053802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:02:57.583889374 +0000 UTC m=+2824.276846767" watchObservedRunningTime="2026-10-31 22:02:57.587053802 +0000 UTC m=+2824.280011197" Oct 31 22:02:57.731153 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:57.731117 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:02:58.733815 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:58.733772 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gjgbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 22:02:58.734231 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:58.733873 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 22:02:59.731469 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:59.731420 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gjgbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 22:02:59.731687 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:02:59.731496 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 22:03:00.731887 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:00.731848 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gjgbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 22:03:00.732367 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:00.731905 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 22:03:01.731303 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:01.731262 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gjgbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 22:03:01.731542 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:01.731332 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 22:03:02.542766 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:03:02.542714 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:03:02.731719 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:02.731671 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gjgbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 22:03:02.731979 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:02.731741 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 22:03:03.731268 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:03.731232 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gjgbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 22:03:03.731815 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:03.731293 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 22:03:04.731276 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:04.731226 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gjgbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 22:03:04.731740 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:04.731300 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 22:03:05.992516 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:05.992480 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:03:05.993048 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:05.992867 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:03:05.994017 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:05.993991 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:03:11.452882 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.452841 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt"] Oct 31 22:03:11.457799 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.457780 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.522587 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.522548 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt"] Oct 31 22:03:11.571764 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.571724 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.571960 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.571872 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.572018 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.572000 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.572079 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.572031 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc92a890-3858-49b3-af22-c1f80c348119-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.572079 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.572066 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc92a890-3858-49b3-af22-c1f80c348119-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.572296 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.572270 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrlx\" (UniqueName: \"kubernetes.io/projected/cc92a890-3858-49b3-af22-c1f80c348119-kube-api-access-9wrlx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.572436 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.572368 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.572513 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.572492 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc92a890-3858-49b3-af22-c1f80c348119-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.572570 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.572534 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674092 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674051 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674092 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674102 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc92a890-3858-49b3-af22-c1f80c348119-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674409 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674136 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc92a890-3858-49b3-af22-c1f80c348119-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674409 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674198 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrlx\" (UniqueName: \"kubernetes.io/projected/cc92a890-3858-49b3-af22-c1f80c348119-kube-api-access-9wrlx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674409 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674248 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674409 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674289 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc92a890-3858-49b3-af22-c1f80c348119-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674409 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674336 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674409 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674380 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674767 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674425 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674767 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674570 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.674767 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.674652 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.675148 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.675116 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.675340 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.675145 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.675453 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.675260 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc92a890-3858-49b3-af22-c1f80c348119-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.676845 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.676820 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc92a890-3858-49b3-af22-c1f80c348119-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.677334 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.677290 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc92a890-3858-49b3-af22-c1f80c348119-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.695738 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.695705 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc92a890-3858-49b3-af22-c1f80c348119-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.696172 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.696148 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrlx\" (UniqueName: \"kubernetes.io/projected/cc92a890-3858-49b3-af22-c1f80c348119-kube-api-access-9wrlx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-btqnt\" (UID: \"cc92a890-3858-49b3-af22-c1f80c348119\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:11.816886 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:11.816787 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:12.156440 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:12.156403 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt"] Oct 31 22:03:12.156806 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:03:12.156783 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc92a890_3858_49b3_af22_c1f80c348119.slice/crio-37aa62e5f93203468f343576a1aa891608cedf5fba25ab708104d19d67499269 WatchSource:0}: Error finding container 37aa62e5f93203468f343576a1aa891608cedf5fba25ab708104d19d67499269: Status 404 returned error can't find the container with id 37aa62e5f93203468f343576a1aa891608cedf5fba25ab708104d19d67499269 Oct 31 22:03:12.159688 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:12.159647 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 22:03:12.159785 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:12.159723 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 22:03:12.159785 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:12.159752 5188 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 22:03:12.703718 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:12.703681 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" event={"ID":"cc92a890-3858-49b3-af22-c1f80c348119","Type":"ContainerStarted","Data":"c29b7799678e1449cf5439c6c670ea1c67b48887593935826e9815e7c7277863"} Oct 31 22:03:12.704252 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:12.703748 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" event={"ID":"cc92a890-3858-49b3-af22-c1f80c348119","Type":"ContainerStarted","Data":"37aa62e5f93203468f343576a1aa891608cedf5fba25ab708104d19d67499269"} Oct 31 22:03:12.765212 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:12.765147 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podStartSLOduration=1.765128909 podStartE2EDuration="1.765128909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:03:12.753380603 +0000 UTC m=+2839.446337996" watchObservedRunningTime="2026-10-31 22:03:12.765128909 +0000 UTC m=+2839.458086301" Oct 31 22:03:12.817246 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:12.817201 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:13.830540 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:13.830493 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-btqnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 22:03:13.831084 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:13.830566 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podUID="cc92a890-3858-49b3-af22-c1f80c348119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 22:03:14.817885 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:14.817848 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-btqnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 22:03:14.818085 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:14.817907 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podUID="cc92a890-3858-49b3-af22-c1f80c348119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 22:03:15.817721 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:15.817688 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-btqnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 22:03:15.818151 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:15.817757 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podUID="cc92a890-3858-49b3-af22-c1f80c348119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 22:03:16.817770 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:16.817731 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-btqnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 22:03:16.818337 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:16.817799 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podUID="cc92a890-3858-49b3-af22-c1f80c348119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 22:03:17.818142 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:17.818101 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-btqnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 22:03:17.818580 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:17.818160 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podUID="cc92a890-3858-49b3-af22-c1f80c348119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 22:03:18.817451 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:18.817403 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-btqnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 22:03:18.817801 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:18.817476 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podUID="cc92a890-3858-49b3-af22-c1f80c348119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 22:03:19.818054 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:19.818012 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-btqnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body= Oct 31 22:03:19.818518 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:19.818087 5188 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podUID="cc92a890-3858-49b3-af22-c1f80c348119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" Oct 31 22:03:21.386927 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:21.386889 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:21.388791 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:21.388761 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:21.389824 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:03:21.389782 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" Oct 31 22:03:32.543242 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:03:32.543194 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:04:02.539998 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:04:02.539886 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:04:32.660592 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:04:32.660524 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:05:02.541690 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:05:02.541643 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:05:32.540928 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:05:32.540887 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:05:54.068054 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:05:54.068022 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:05:54.090065 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:05:54.090034 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:05:57.457847 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:05:57.457808 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-k4qxr_e49a2d50-dc82-4983-9425-053dfe903ddd/cluster-samples-operator/0.log" Oct 31 22:05:57.458282 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:05:57.457851 5188 generic.go:356] "Generic (PLEG): container finished" podID="e49a2d50-dc82-4983-9425-053dfe903ddd" containerID="82cefe810f98c18337fe0482d7e6506abefe0a7091297bc9899cf5101a3a6dce" exitCode=2 Oct 31 22:05:57.458282 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:05:57.457926 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" event={"ID":"e49a2d50-dc82-4983-9425-053dfe903ddd","Type":"ContainerDied","Data":"82cefe810f98c18337fe0482d7e6506abefe0a7091297bc9899cf5101a3a6dce"} Oct 31 22:05:57.458421 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:05:57.458406 5188 scope.go:122] "RemoveContainer" containerID="82cefe810f98c18337fe0482d7e6506abefe0a7091297bc9899cf5101a3a6dce" Oct 31 22:05:58.469530 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:05:58.469499 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-k4qxr_e49a2d50-dc82-4983-9425-053dfe903ddd/cluster-samples-operator/0.log" Oct 31 22:05:58.469949 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:05:58.469618 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-k4qxr" event={"ID":"e49a2d50-dc82-4983-9425-053dfe903ddd","Type":"ContainerStarted","Data":"441c848e1b36fc5ac03ead550784ec862180bbc0d34e966c43e0de66334ea84b"} Oct 31 22:06:02.540042 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:06:02.539998 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:06:32.539599 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:06:32.539498 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:07:02.538984 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:07:02.538943 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:07:24.767313 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:24.767270 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm"] Oct 31 22:07:24.767931 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:24.767529 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" containerID="cri-o://79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9" gracePeriod=30 Oct 31 22:07:30.064629 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.064595 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 22:07:30.096303 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096255 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-certs\") pod \"80080a1d-b279-4bab-a2da-48c1cd251701\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " Oct 31 22:07:30.096610 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096427 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-kube-api-access-7vjp6\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-kube-api-access-7vjp6\") pod \"80080a1d-b279-4bab-a2da-48c1cd251701\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " Oct 31 22:07:30.096610 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096480 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80080a1d-b279-4bab-a2da-48c1cd251701-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80080a1d-b279-4bab-a2da-48c1cd251701-istio-podinfo\") pod \"80080a1d-b279-4bab-a2da-48c1cd251701\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " Oct 31 22:07:30.096610 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096561 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80080a1d-b279-4bab-a2da-48c1cd251701-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80080a1d-b279-4bab-a2da-48c1cd251701-istiod-ca-cert\") pod \"80080a1d-b279-4bab-a2da-48c1cd251701\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " Oct 31 22:07:30.096610 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096582 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-certs" pod "80080a1d-b279-4bab-a2da-48c1cd251701" (UID: "80080a1d-b279-4bab-a2da-48c1cd251701"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:30.097001 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096591 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-socket\") pod \"80080a1d-b279-4bab-a2da-48c1cd251701\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " Oct 31 22:07:30.097001 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096658 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-istio-token\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-istio-token\") pod \"80080a1d-b279-4bab-a2da-48c1cd251701\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " Oct 31 22:07:30.097001 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096716 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-envoy\") pod \"80080a1d-b279-4bab-a2da-48c1cd251701\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " Oct 31 22:07:30.097001 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096752 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-credential-socket\") pod \"80080a1d-b279-4bab-a2da-48c1cd251701\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " Oct 31 22:07:30.097001 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096826 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-data\") pod \"80080a1d-b279-4bab-a2da-48c1cd251701\" (UID: \"80080a1d-b279-4bab-a2da-48c1cd251701\") " Oct 31 22:07:30.097001 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.096853 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-socket" pod "80080a1d-b279-4bab-a2da-48c1cd251701" (UID: "80080a1d-b279-4bab-a2da-48c1cd251701"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:30.097921 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.097462 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:07:30.097921 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.097481 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:07:30.097921 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.097683 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-data" pod "80080a1d-b279-4bab-a2da-48c1cd251701" (UID: "80080a1d-b279-4bab-a2da-48c1cd251701"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:30.097921 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.097739 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80080a1d-b279-4bab-a2da-48c1cd251701-istiod-ca-cert" pod "80080a1d-b279-4bab-a2da-48c1cd251701" (UID: "80080a1d-b279-4bab-a2da-48c1cd251701"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:30.097921 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.097848 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-credential-socket" pod "80080a1d-b279-4bab-a2da-48c1cd251701" (UID: "80080a1d-b279-4bab-a2da-48c1cd251701"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:30.099643 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.099559 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-kube-api-access-7vjp6" pod "80080a1d-b279-4bab-a2da-48c1cd251701" (UID: "80080a1d-b279-4bab-a2da-48c1cd251701"). InnerVolumeSpecName "kube-api-access-7vjp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:30.099824 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.099788 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-envoy" pod "80080a1d-b279-4bab-a2da-48c1cd251701" (UID: "80080a1d-b279-4bab-a2da-48c1cd251701"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:30.100108 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.100081 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-istio-token" pod "80080a1d-b279-4bab-a2da-48c1cd251701" (UID: "80080a1d-b279-4bab-a2da-48c1cd251701"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:30.100634 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.100606 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80080a1d-b279-4bab-a2da-48c1cd251701-istio-podinfo" pod "80080a1d-b279-4bab-a2da-48c1cd251701" (UID: "80080a1d-b279-4bab-a2da-48c1cd251701"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:07:30.198863 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.198822 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:07:30.198863 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.198860 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vjp6\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-kube-api-access-7vjp6\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:07:30.199126 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.198873 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80080a1d-b279-4bab-a2da-48c1cd251701-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:07:30.199126 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.198885 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80080a1d-b279-4bab-a2da-48c1cd251701-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:07:30.199126 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.198896 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80080a1d-b279-4bab-a2da-48c1cd251701-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:07:30.199126 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.198906 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:07:30.199126 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.198917 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80080a1d-b279-4bab-a2da-48c1cd251701-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:07:30.395665 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.395632 5188 generic.go:356] "Generic (PLEG): container finished" podID="80080a1d-b279-4bab-a2da-48c1cd251701" containerID="79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9" exitCode=0 Oct 31 22:07:30.395850 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.395706 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" Oct 31 22:07:30.395850 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.395720 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" event={"ID":"80080a1d-b279-4bab-a2da-48c1cd251701","Type":"ContainerDied","Data":"79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9"} Oct 31 22:07:30.395850 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.395778 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm" event={"ID":"80080a1d-b279-4bab-a2da-48c1cd251701","Type":"ContainerDied","Data":"85932a20692a889701ee3329b5f37c005d50cdce0bb3fbf442ed9cbc9f9ebaa1"} Oct 31 22:07:30.395850 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.395804 5188 scope.go:122] "RemoveContainer" containerID="79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9" Oct 31 22:07:30.408600 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.408577 5188 scope.go:122] "RemoveContainer" containerID="79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9" Oct 31 22:07:30.408962 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:07:30.408937 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9\": container with ID starting with 79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9 not found: ID does not exist" containerID="79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9" Oct 31 22:07:30.409077 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.408968 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9"} err="failed to get container status \"79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9\": rpc error: code = NotFound desc = could not find container \"79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9\": container with ID starting with 79f8a7667bae128d61c339a31566b591860406528755c3ab04c8f2330d8dc4f9 not found: ID does not exist" Oct 31 22:07:30.442733 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.442696 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm"] Oct 31 22:07:30.452393 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:30.452355 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-pq6gm"] Oct 31 22:07:31.504047 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:07:31.502953 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" path="/var/lib/kubelet/pods/80080a1d-b279-4bab-a2da-48c1cd251701/volumes" Oct 31 22:07:32.540236 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:07:32.540191 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:08:02.541148 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:08:02.541103 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:08:07.042348 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:07.041448 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gjgbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:08:07.042348 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:07.041528 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:08:07.138349 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:07.137308 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vxrb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:08:07.138349 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:07.137417 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:08:07.439728 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:07.439602 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-btqnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:08:07.439728 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:07.439677 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podUID="cc92a890-3858-49b3-af22-c1f80c348119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:08:31.407206 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.407154 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7"] Oct 31 22:08:31.408194 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.408170 5188 cpu_manager.go:414] "containerMap: removing container" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" Oct 31 22:08:31.408194 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.408196 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" Oct 31 22:08:31.408344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.408332 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="80080a1d-b279-4bab-a2da-48c1cd251701" containerName="istio-proxy" Oct 31 22:08:31.412158 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.412138 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.434965 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.434931 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-710927-serving-cert\"" Oct 31 22:08:31.439790 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.439754 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7"] Oct 31 22:08:31.541387 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.541342 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55j8g\" (UniqueName: \"kubernetes.io/projected/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-kube-api-access-55j8g\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.541608 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.541483 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-metrics-certs\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.541693 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.541601 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-maas-api-tls\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.541866 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.541846 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-openshift-service-ca\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.657617 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.657514 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55j8g\" (UniqueName: \"kubernetes.io/projected/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-kube-api-access-55j8g\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.672349 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.663457 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-metrics-certs\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.674353 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.673778 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-maas-api-tls\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.695800 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.694033 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-openshift-service-ca\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.695800 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.695104 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-openshift-service-ca\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.705557 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.698777 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-maas-api-tls\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.705557 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.699188 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-metrics-certs\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.713314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.713279 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55j8g\" (UniqueName: \"kubernetes.io/projected/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-kube-api-access-55j8g\") pod \"maas-api-e2e-inf-a-710927-7d44765d85-7qrf7\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:31.727971 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:31.727929 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:32.054665 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:32.054634 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7"] Oct 31 22:08:32.055031 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:08:32.055003 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853d5e9a_0ff1_4dd9_be7c_efda8dfaa456.slice/crio-4b8f3aab8b5a306268c2adfce4d350241a103ff48960f7aa81cc14c794479969 WatchSource:0}: Error finding container 4b8f3aab8b5a306268c2adfce4d350241a103ff48960f7aa81cc14c794479969: Status 404 returned error can't find the container with id 4b8f3aab8b5a306268c2adfce4d350241a103ff48960f7aa81cc14c794479969 Oct 31 22:08:32.056418 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:32.056399 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 22:08:32.640782 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:08:32.640739 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:08:33.019946 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:33.019900 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" event={"ID":"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456","Type":"ContainerStarted","Data":"7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47"} Oct 31 22:08:33.020130 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:33.019971 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:08:33.020130 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:33.019993 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" event={"ID":"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456","Type":"ContainerStarted","Data":"4b8f3aab8b5a306268c2adfce4d350241a103ff48960f7aa81cc14c794479969"} Oct 31 22:08:33.060059 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:33.060001 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" podStartSLOduration=1.639872558 podStartE2EDuration="2.059985267s" totalImagesPullingTime="420.112709ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:08:33.057335162 +0000 UTC m=+3159.750292550" watchObservedRunningTime="2026-10-31 22:08:33.059985267 +0000 UTC m=+3159.752942661" Oct 31 22:08:39.033473 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:08:39.033442 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:09:02.545158 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:09:02.545023 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:09:32.541738 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:09:32.541594 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:09:33.964888 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:33.964848 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27"] Oct 31 22:09:33.965622 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:33.965075 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" podUID="8cb5b051-50c9-4a83-a6d5-d49c87a93b07" containerName="maas-api" containerID="cri-o://1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede" gracePeriod=30 Oct 31 22:09:34.283347 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.283093 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 22:09:34.368344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.365990 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-maas-api-tls\") pod \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " Oct 31 22:09:34.368344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.366077 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-openshift-service-ca\") pod \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " Oct 31 22:09:34.368344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.366135 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-kube-api-access-jpqc4\" (UniqueName: \"kubernetes.io/projected/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-kube-api-access-jpqc4\") pod \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " Oct 31 22:09:34.368344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.366162 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-metrics-certs\") pod \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\" (UID: \"8cb5b051-50c9-4a83-a6d5-d49c87a93b07\") " Oct 31 22:09:34.368344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.367374 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-openshift-service-ca" pod "8cb5b051-50c9-4a83-a6d5-d49c87a93b07" (UID: "8cb5b051-50c9-4a83-a6d5-d49c87a93b07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:34.392357 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.391616 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-maas-api-tls" pod "8cb5b051-50c9-4a83-a6d5-d49c87a93b07" (UID: "8cb5b051-50c9-4a83-a6d5-d49c87a93b07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:09:34.435830 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.423713 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-metrics-certs" pod "8cb5b051-50c9-4a83-a6d5-d49c87a93b07" (UID: "8cb5b051-50c9-4a83-a6d5-d49c87a93b07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:09:34.469623 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.469520 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:09:34.469623 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.469557 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:09:34.469623 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.469573 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:09:34.469929 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.469766 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-kube-api-access-jpqc4" pod "8cb5b051-50c9-4a83-a6d5-d49c87a93b07" (UID: "8cb5b051-50c9-4a83-a6d5-d49c87a93b07"). InnerVolumeSpecName "kube-api-access-jpqc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:34.570971 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.570940 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqc4\" (UniqueName: \"kubernetes.io/projected/8cb5b051-50c9-4a83-a6d5-d49c87a93b07-kube-api-access-jpqc4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:09:34.583341 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.583286 5188 generic.go:356] "Generic (PLEG): container finished" podID="8cb5b051-50c9-4a83-a6d5-d49c87a93b07" containerID="1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede" exitCode=0 Oct 31 22:09:34.583524 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.583460 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" event={"ID":"8cb5b051-50c9-4a83-a6d5-d49c87a93b07","Type":"ContainerDied","Data":"1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede"} Oct 31 22:09:34.583524 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.583500 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" event={"ID":"8cb5b051-50c9-4a83-a6d5-d49c87a93b07","Type":"ContainerDied","Data":"66563a692fbb73c162b52e733fe21de166d387be5399a942b9882fe089cad781"} Oct 31 22:09:34.583651 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.583520 5188 scope.go:122] "RemoveContainer" containerID="1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede" Oct 31 22:09:34.583704 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.583690 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27" Oct 31 22:09:34.613725 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.613689 5188 scope.go:122] "RemoveContainer" containerID="1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede" Oct 31 22:09:34.614450 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:09:34.614354 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede\": container with ID starting with 1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede not found: ID does not exist" containerID="1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede" Oct 31 22:09:34.614450 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.614400 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede"} err="failed to get container status \"1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede\": rpc error: code = NotFound desc = could not find container \"1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede\": container with ID starting with 1fc62f89ba3c08d6a7f0322c6656114e1148d103efdf466f51d1af57105d9ede not found: ID does not exist" Oct 31 22:09:34.658380 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.658340 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27"] Oct 31 22:09:34.668824 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:34.668777 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fac043-645b8f57b-ptt27"] Oct 31 22:09:35.483605 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:35.483528 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb5b051-50c9-4a83-a6d5-d49c87a93b07" path="/var/lib/kubelet/pods/8cb5b051-50c9-4a83-a6d5-d49c87a93b07/volumes" Oct 31 22:09:37.049345 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:37.048532 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gjgbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:09:37.049345 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:37.048630 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:09:37.150399 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:37.143452 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-vxrb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:09:37.150399 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:37.143567 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:09:37.453348 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:37.452491 5188 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-btqnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:09:37.453348 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:37.452643 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-btqnt" podUID="cc92a890-3858-49b3-af22-c1f80c348119" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:09:55.764970 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.764928 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5"] Oct 31 22:09:55.765845 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.765818 5188 cpu_manager.go:414] "containerMap: removing container" podUID="8cb5b051-50c9-4a83-a6d5-d49c87a93b07" containerName="maas-api" Oct 31 22:09:55.765845 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.765846 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb5b051-50c9-4a83-a6d5-d49c87a93b07" containerName="maas-api" Oct 31 22:09:55.765989 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.765978 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb5b051-50c9-4a83-a6d5-d49c87a93b07" containerName="maas-api" Oct 31 22:09:55.769721 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.769697 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:55.784836 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.784802 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-cd421c-serving-cert\"" Oct 31 22:09:55.799608 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.799566 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5"] Oct 31 22:09:55.898844 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.898794 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-metrics-certs\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:55.899083 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.899018 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:55.899165 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.899088 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7ks\" (UniqueName: \"kubernetes.io/projected/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-kube-api-access-wr7ks\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:55.899227 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:55.899178 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-maas-api-tls\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.002997 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.002950 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.004245 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.004218 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7ks\" (UniqueName: \"kubernetes.io/projected/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-kube-api-access-wr7ks\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.004805 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.004781 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-maas-api-tls\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.005057 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.005037 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-metrics-certs\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.007458 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.004086 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-openshift-service-ca\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.012436 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.012020 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-maas-api-tls\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.018413 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.015174 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-metrics-certs\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.029086 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.029050 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7ks\" (UniqueName: \"kubernetes.io/projected/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-kube-api-access-wr7ks\") pod \"maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.088309 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.088272 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:56.399854 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.399818 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5"] Oct 31 22:09:56.401146 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:09:56.401101 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b91cc9_b3f0_4fff_b1f3_049664e40ea2.slice/crio-c19ccc5348fefab793647b578148eac7bbdb271d1793afb02bf1dbfed0a6dc40 WatchSource:0}: Error finding container c19ccc5348fefab793647b578148eac7bbdb271d1793afb02bf1dbfed0a6dc40: Status 404 returned error can't find the container with id c19ccc5348fefab793647b578148eac7bbdb271d1793afb02bf1dbfed0a6dc40 Oct 31 22:09:56.834119 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:56.834077 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" event={"ID":"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2","Type":"ContainerStarted","Data":"c19ccc5348fefab793647b578148eac7bbdb271d1793afb02bf1dbfed0a6dc40"} Oct 31 22:09:57.843568 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:57.843525 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" event={"ID":"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2","Type":"ContainerStarted","Data":"4a7141e85dbf1fdba535bde2548193d88e2cec8cde6dad1210d505c3a04b1100"} Oct 31 22:09:57.844040 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:57.843615 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:09:57.885121 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:09:57.885060 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" podStartSLOduration=2.521439838 podStartE2EDuration="2.885041662s" totalImagesPullingTime="363.601824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:57.883955856 +0000 UTC m=+3244.576913250" watchObservedRunningTime="2026-10-31 22:09:57.885041662 +0000 UTC m=+3244.577999054" Oct 31 22:10:02.540240 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:10:02.540199 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:10:03.856548 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:03.856521 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:10:06.007434 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.007393 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm"] Oct 31 22:10:06.014160 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.014129 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.032942 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.032901 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm"] Oct 31 22:10:06.034488 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.034404 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c2d1b8-serving-cert\"" Oct 31 22:10:06.078220 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.078156 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdhz\" (UniqueName: \"kubernetes.io/projected/68f1815c-225d-4141-9279-eb765de32aa3-kube-api-access-7hdhz\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.078780 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.078753 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f1815c-225d-4141-9279-eb765de32aa3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.079002 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.078986 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-metrics-certs\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.079171 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.079156 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-maas-api-tls\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.197364 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.187399 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f1815c-225d-4141-9279-eb765de32aa3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.197906 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.197851 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-metrics-certs\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.202208 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.202165 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-maas-api-tls\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.202818 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.202657 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdhz\" (UniqueName: \"kubernetes.io/projected/68f1815c-225d-4141-9279-eb765de32aa3-kube-api-access-7hdhz\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.211416 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.195551 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f1815c-225d-4141-9279-eb765de32aa3-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.224089 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.224048 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-metrics-certs\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.225193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.225067 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-maas-api-tls\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.239343 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.239281 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdhz\" (UniqueName: \"kubernetes.io/projected/68f1815c-225d-4141-9279-eb765de32aa3-kube-api-access-7hdhz\") pod \"maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.337029 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.336927 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:06.651736 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.651686 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm"] Oct 31 22:10:06.652379 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:10:06.652352 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f1815c_225d_4141_9279_eb765de32aa3.slice/crio-a66bc3cf8a2b2d9bbb18e2a5b12f6f9c5583aea9ddc00d44ffc0b50b8b5eb9d8 WatchSource:0}: Error finding container a66bc3cf8a2b2d9bbb18e2a5b12f6f9c5583aea9ddc00d44ffc0b50b8b5eb9d8: Status 404 returned error can't find the container with id a66bc3cf8a2b2d9bbb18e2a5b12f6f9c5583aea9ddc00d44ffc0b50b8b5eb9d8 Oct 31 22:10:06.927642 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:06.927552 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" event={"ID":"68f1815c-225d-4141-9279-eb765de32aa3","Type":"ContainerStarted","Data":"a66bc3cf8a2b2d9bbb18e2a5b12f6f9c5583aea9ddc00d44ffc0b50b8b5eb9d8"} Oct 31 22:10:07.971281 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:07.971237 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" event={"ID":"68f1815c-225d-4141-9279-eb765de32aa3","Type":"ContainerStarted","Data":"040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e"} Oct 31 22:10:07.972420 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:07.972392 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:08.016347 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:08.014215 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" podStartSLOduration=2.640609884 podStartE2EDuration="3.014120127s" totalImagesPullingTime="373.510243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:10:08.010601054 +0000 UTC m=+3254.703558441" watchObservedRunningTime="2026-10-31 22:10:08.014120127 +0000 UTC m=+3254.707077522" Oct 31 22:10:14.993605 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:14.993558 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:10:26.601125 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:26.601079 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9"] Oct 31 22:10:26.601838 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:26.601404 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" podUID="10512159-6a38-4a9d-b7d0-fd3f7aaace1c" containerName="maas-api" containerID="cri-o://f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4" gracePeriod=30 Oct 31 22:10:27.025350 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.019599 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 22:10:27.109375 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.089219 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-kube-api-access-4rtfs\" (UniqueName: \"kubernetes.io/projected/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-kube-api-access-4rtfs\") pod \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " Oct 31 22:10:27.109375 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.089372 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-maas-api-tls\") pod \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " Oct 31 22:10:27.109375 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.089488 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-metrics-certs\") pod \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " Oct 31 22:10:27.109375 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.089575 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-openshift-service-ca\") pod \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\" (UID: \"10512159-6a38-4a9d-b7d0-fd3f7aaace1c\") " Oct 31 22:10:27.109375 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.096746 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-openshift-service-ca" pod "10512159-6a38-4a9d-b7d0-fd3f7aaace1c" (UID: "10512159-6a38-4a9d-b7d0-fd3f7aaace1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:10:27.109375 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.103287 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-maas-api-tls" pod "10512159-6a38-4a9d-b7d0-fd3f7aaace1c" (UID: "10512159-6a38-4a9d-b7d0-fd3f7aaace1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:10:27.119349 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.116454 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-metrics-certs" pod "10512159-6a38-4a9d-b7d0-fd3f7aaace1c" (UID: "10512159-6a38-4a9d-b7d0-fd3f7aaace1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:10:27.132361 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.126543 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-kube-api-access-4rtfs" pod "10512159-6a38-4a9d-b7d0-fd3f7aaace1c" (UID: "10512159-6a38-4a9d-b7d0-fd3f7aaace1c"). InnerVolumeSpecName "kube-api-access-4rtfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:10:27.158301 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.158257 5188 generic.go:356] "Generic (PLEG): container finished" podID="10512159-6a38-4a9d-b7d0-fd3f7aaace1c" containerID="f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4" exitCode=0 Oct 31 22:10:27.159897 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.158472 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" event={"ID":"10512159-6a38-4a9d-b7d0-fd3f7aaace1c","Type":"ContainerDied","Data":"f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4"} Oct 31 22:10:27.159897 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.158540 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" event={"ID":"10512159-6a38-4a9d-b7d0-fd3f7aaace1c","Type":"ContainerDied","Data":"fad76cea3383f304761e3e951e14ab2733f33530279cadc8efc1e68ee108fd17"} Oct 31 22:10:27.159897 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.158582 5188 scope.go:122] "RemoveContainer" containerID="f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4" Oct 31 22:10:27.159897 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.158776 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9" Oct 31 22:10:27.181119 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.180759 5188 scope.go:122] "RemoveContainer" containerID="f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4" Oct 31 22:10:27.192583 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:10:27.192537 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4\": container with ID starting with f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4 not found: ID does not exist" containerID="f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4" Oct 31 22:10:27.192762 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.192592 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4"} err="failed to get container status \"f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4\": rpc error: code = NotFound desc = could not find container \"f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4\": container with ID starting with f5033f66e7933e6dcdbbc091ee2443347011b4c5c9ca1e3d4dac2392052ec0f4 not found: ID does not exist" Oct 31 22:10:27.195007 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.194972 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:10:27.195155 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.195010 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:10:27.195155 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.195026 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rtfs\" (UniqueName: \"kubernetes.io/projected/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-kube-api-access-4rtfs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:10:27.195155 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.195040 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10512159-6a38-4a9d-b7d0-fd3f7aaace1c-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:10:27.218369 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.218268 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9"] Oct 31 22:10:27.224219 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.224191 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2dbae9-5d79c7f845-gr7k9"] Oct 31 22:10:27.481718 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:27.481610 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10512159-6a38-4a9d-b7d0-fd3f7aaace1c" path="/var/lib/kubelet/pods/10512159-6a38-4a9d-b7d0-fd3f7aaace1c/volumes" Oct 31 22:10:32.540266 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:10:32.540224 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:10:54.134442 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:54.134409 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-k4qxr_e49a2d50-dc82-4983-9425-053dfe903ddd/cluster-samples-operator/0.log" Oct 31 22:10:54.137453 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:54.137421 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:10:54.153880 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:54.153851 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-k4qxr_e49a2d50-dc82-4983-9425-053dfe903ddd/cluster-samples-operator/0.log" Oct 31 22:10:54.156812 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:10:54.156783 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:11:02.540935 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:11:02.540895 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:11:12.038966 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.038921 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t"] Oct 31 22:11:12.039674 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.039207 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" podUID="ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" containerName="maas-api" containerID="cri-o://b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26" gracePeriod=30 Oct 31 22:11:12.523388 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.517555 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 22:11:12.634392 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.634310 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-openshift-service-ca\") pod \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " Oct 31 22:11:12.634589 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.634444 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-kube-api-access-w4zcq\") pod \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " Oct 31 22:11:12.634647 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.634610 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-maas-api-tls\") pod \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " Oct 31 22:11:12.634703 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.634678 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-openshift-service-ca" pod "ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" (UID: "ac5bbbce-bc34-4a76-b3d8-ed871fa499a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:12.634767 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.634736 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-metrics-certs\") pod \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\" (UID: \"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7\") " Oct 31 22:11:12.635481 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.635455 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:12.636995 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.636969 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-maas-api-tls" pod "ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" (UID: "ac5bbbce-bc34-4a76-b3d8-ed871fa499a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:11:12.637315 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.637284 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-kube-api-access-w4zcq" pod "ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" (UID: "ac5bbbce-bc34-4a76-b3d8-ed871fa499a7"). InnerVolumeSpecName "kube-api-access-w4zcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:12.637454 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.637376 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-metrics-certs" pod "ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" (UID: "ac5bbbce-bc34-4a76-b3d8-ed871fa499a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:11:12.736864 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.736823 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:12.736864 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.736863 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:12.737114 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.736877 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4zcq\" (UniqueName: \"kubernetes.io/projected/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7-kube-api-access-w4zcq\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:12.761071 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.761033 5188 generic.go:356] "Generic (PLEG): container finished" podID="ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" containerID="b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26" exitCode=0 Oct 31 22:11:12.761283 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.761079 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" Oct 31 22:11:12.761283 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.761118 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" event={"ID":"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7","Type":"ContainerDied","Data":"b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26"} Oct 31 22:11:12.761283 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.761173 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t" event={"ID":"ac5bbbce-bc34-4a76-b3d8-ed871fa499a7","Type":"ContainerDied","Data":"c4d2763a35f0498155e466655db34e6c7671ec483567faa7e3ec0bc012107725"} Oct 31 22:11:12.761283 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.761225 5188 scope.go:122] "RemoveContainer" containerID="b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26" Oct 31 22:11:12.773913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.773819 5188 scope.go:122] "RemoveContainer" containerID="b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26" Oct 31 22:11:12.774178 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:11:12.774158 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26\": container with ID starting with b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26 not found: ID does not exist" containerID="b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26" Oct 31 22:11:12.774248 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.774184 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26"} err="failed to get container status \"b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26\": rpc error: code = NotFound desc = could not find container \"b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26\": container with ID starting with b95647f7a0530c4fbaa774b99be0f5d260021f57e026187219ec2f656cb46b26 not found: ID does not exist" Oct 31 22:11:12.813434 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.813394 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t"] Oct 31 22:11:12.817095 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:12.817066 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0877f5-647dff899d-qsq4t"] Oct 31 22:11:13.489084 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:13.489046 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" path="/var/lib/kubelet/pods/ac5bbbce-bc34-4a76-b3d8-ed871fa499a7/volumes" Oct 31 22:11:18.137312 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.137268 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4"] Oct 31 22:11:18.138414 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.138389 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" containerName="maas-api" Oct 31 22:11:18.138414 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.138413 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" containerName="maas-api" Oct 31 22:11:18.138619 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.138432 5188 cpu_manager.go:414] "containerMap: removing container" podUID="10512159-6a38-4a9d-b7d0-fd3f7aaace1c" containerName="maas-api" Oct 31 22:11:18.138619 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.138441 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10512159-6a38-4a9d-b7d0-fd3f7aaace1c" containerName="maas-api" Oct 31 22:11:18.138729 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.138640 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5bbbce-bc34-4a76-b3d8-ed871fa499a7" containerName="maas-api" Oct 31 22:11:18.138729 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.138653 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="10512159-6a38-4a9d-b7d0-fd3f7aaace1c" containerName="maas-api" Oct 31 22:11:18.144481 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.144447 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.162659 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.162620 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4"] Oct 31 22:11:18.166639 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.166611 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c2d1b8\"/\"default-dockercfg-cl58p\"" Oct 31 22:11:18.166952 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.166612 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c2d1b8\"/\"test-model-c2d1b8-kserve-self-signed-certs\"" Oct 31 22:11:18.167820 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.167795 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c2d1b8\"/\"kube-root-ca.crt\"" Oct 31 22:11:18.167985 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.167854 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c2d1b8\"/\"openshift-service-ca.crt\"" Oct 31 22:11:18.343261 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.343223 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tmp-dir\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.343500 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.343288 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-dshm\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.343500 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.343459 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tls-certs\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.343653 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.343616 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-home\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.343839 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.343815 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-model-cache\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.343983 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.343863 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwcdn\" (UniqueName: \"kubernetes.io/projected/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-kube-api-access-wwcdn\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.444982 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.444891 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-dshm\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.444982 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.444937 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tls-certs\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.445259 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.445004 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-home\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.445259 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.445172 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-model-cache\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.445259 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.445215 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwcdn\" (UniqueName: \"kubernetes.io/projected/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-kube-api-access-wwcdn\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.445470 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.445276 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tmp-dir\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.445541 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.445459 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-home\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.445599 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.445536 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-model-cache\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.445658 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.445609 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tmp-dir\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.447459 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.447430 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-dshm\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.447699 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.447681 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tls-certs\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.466803 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.466765 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwcdn\" (UniqueName: \"kubernetes.io/projected/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-kube-api-access-wwcdn\") pod \"test-model-c2d1b8-kserve-db65b8447-jlpr4\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:18.762472 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:18.762319 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:19.080411 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:19.080374 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4"] Oct 31 22:11:19.081543 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:11:19.081515 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb87ebb5_bb07_4598_b6e0_c3d587a05cbf.slice/crio-8d400d6d25fa1f4b5d5e99c5bceb2060601c61eb866a6fb9b5736eacaaa053dc WatchSource:0}: Error finding container 8d400d6d25fa1f4b5d5e99c5bceb2060601c61eb866a6fb9b5736eacaaa053dc: Status 404 returned error can't find the container with id 8d400d6d25fa1f4b5d5e99c5bceb2060601c61eb866a6fb9b5736eacaaa053dc Oct 31 22:11:19.823395 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:19.823345 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" event={"ID":"db87ebb5-bb07-4598-b6e0-c3d587a05cbf","Type":"ContainerStarted","Data":"f0f2d5c0c64cc2f8982cc92da47a892b409a4a0daa6904b39c3da556fb51dc96"} Oct 31 22:11:19.823841 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:19.823406 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" event={"ID":"db87ebb5-bb07-4598-b6e0-c3d587a05cbf","Type":"ContainerStarted","Data":"8d400d6d25fa1f4b5d5e99c5bceb2060601c61eb866a6fb9b5736eacaaa053dc"} Oct 31 22:11:19.861020 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:19.860956 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" podStartSLOduration=1.860941236 podStartE2EDuration="1.860941236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:11:19.859498853 +0000 UTC m=+3326.552456246" watchObservedRunningTime="2026-10-31 22:11:19.860941236 +0000 UTC m=+3326.553898629" Oct 31 22:11:20.762953 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:20.762895 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:28.763506 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:28.763466 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:28.776849 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:28.776814 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:28.914231 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:28.914200 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:11:31.447233 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:31.447190 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n"] Oct 31 22:11:31.486955 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:31.486913 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz"] Oct 31 22:11:32.071233 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.071195 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2"] Oct 31 22:11:32.076840 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.076804 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.151348 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.123476 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c08360-serving-cert\"" Oct 31 22:11:32.162243 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.162197 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6lf\" (UniqueName: \"kubernetes.io/projected/a039b42c-216c-433c-9a1c-a200e9ab5374-kube-api-access-2m6lf\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.162652 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.162620 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-metrics-certs\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.162807 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.162790 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.163036 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.163017 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a039b42c-216c-433c-9a1c-a200e9ab5374-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.179345 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.169076 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2"] Oct 31 22:11:32.275050 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.274990 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a039b42c-216c-433c-9a1c-a200e9ab5374-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.277031 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.276843 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a039b42c-216c-433c-9a1c-a200e9ab5374-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.277476 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.277450 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6lf\" (UniqueName: \"kubernetes.io/projected/a039b42c-216c-433c-9a1c-a200e9ab5374-kube-api-access-2m6lf\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.278290 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.278262 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-metrics-certs\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.278976 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.278956 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.279503 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:11:32.279231 5188 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-serving-cert: secret "maas-api-e2e-inf-b-c08360-serving-cert" not found Oct 31 22:11:32.279759 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:11:32.279737 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls podName:a039b42c-216c-433c-9a1c-a200e9ab5374 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:32.779708832 +0000 UTC m=+3339.472666243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls") pod "maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" (UID: "a039b42c-216c-433c-9a1c-a200e9ab5374") : secret "maas-api-e2e-inf-b-c08360-serving-cert" not found Oct 31 22:11:32.311313 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.304996 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-metrics-certs\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.321433 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.320021 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6lf\" (UniqueName: \"kubernetes.io/projected/a039b42c-216c-433c-9a1c-a200e9ab5374-kube-api-access-2m6lf\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.541298 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:11:32.541253 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:11:32.786904 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.786859 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.789934 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.789895 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls\") pod \"maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:32.821902 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:32.821860 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:33.135819 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:33.135771 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2"] Oct 31 22:11:33.136573 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:11:33.136539 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda039b42c_216c_433c_9a1c_a200e9ab5374.slice/crio-3ca63bec088ce7ed2bbd3d9ee866eec9aa1291659ef22bfbcfe17591ea0d3022 WatchSource:0}: Error finding container 3ca63bec088ce7ed2bbd3d9ee866eec9aa1291659ef22bfbcfe17591ea0d3022: Status 404 returned error can't find the container with id 3ca63bec088ce7ed2bbd3d9ee866eec9aa1291659ef22bfbcfe17591ea0d3022 Oct 31 22:11:33.976004 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:33.975960 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" event={"ID":"a039b42c-216c-433c-9a1c-a200e9ab5374","Type":"ContainerStarted","Data":"96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a"} Oct 31 22:11:33.976583 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:33.976064 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" event={"ID":"a039b42c-216c-433c-9a1c-a200e9ab5374","Type":"ContainerStarted","Data":"3ca63bec088ce7ed2bbd3d9ee866eec9aa1291659ef22bfbcfe17591ea0d3022"} Oct 31 22:11:33.976583 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:33.976098 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:34.050833 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:34.050761 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" podStartSLOduration=1.619763899 podStartE2EDuration="2.050738212s" totalImagesPullingTime="430.974313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:11:34.038900615 +0000 UTC m=+3340.731858023" watchObservedRunningTime="2026-10-31 22:11:34.050738212 +0000 UTC m=+3340.743695607" Oct 31 22:11:36.448002 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:36.447926 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" podUID="6668ee01-98ae-4bbc-b134-3df50eeeb23f" containerName="payload-processing" containerID="cri-o://463ded2a1846fde73af96e4f4732ba44d3f7ecff84e0ddee5d5f9e6946489c80" gracePeriod=25 Oct 31 22:11:36.488089 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:36.488045 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" podUID="83480925-f78a-400c-83f5-6d44260c7d3c" containerName="payload-pre-processing" containerID="cri-o://a8fc7227f12f63e601b188183b1c75936ead110da58ecd6611171362a6395084" gracePeriod=25 Oct 31 22:11:37.529761 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:37.529714 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" podUID="6668ee01-98ae-4bbc-b134-3df50eeeb23f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:11:37.530221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:37.529800 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" podUID="83480925-f78a-400c-83f5-6d44260c7d3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:11:38.583982 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:38.583943 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5"] Oct 31 22:11:38.584513 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:38.584194 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" podUID="17c8a256-3038-419a-b2df-fbf1991bfe30" containerName="maas-api" containerID="cri-o://4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493" gracePeriod=30 Oct 31 22:11:38.963847 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:38.963815 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 22:11:38.986349 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:38.985790 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-metrics-certs\") pod \"17c8a256-3038-419a-b2df-fbf1991bfe30\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " Oct 31 22:11:38.986349 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:38.985919 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c8a256-3038-419a-b2df-fbf1991bfe30-kube-api-access-mxmfd\" (UniqueName: \"kubernetes.io/projected/17c8a256-3038-419a-b2df-fbf1991bfe30-kube-api-access-mxmfd\") pod \"17c8a256-3038-419a-b2df-fbf1991bfe30\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " Oct 31 22:11:38.986349 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:38.986066 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-maas-api-tls\") pod \"17c8a256-3038-419a-b2df-fbf1991bfe30\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " Oct 31 22:11:38.986349 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:38.986152 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c8a256-3038-419a-b2df-fbf1991bfe30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8a256-3038-419a-b2df-fbf1991bfe30-openshift-service-ca\") pod \"17c8a256-3038-419a-b2df-fbf1991bfe30\" (UID: \"17c8a256-3038-419a-b2df-fbf1991bfe30\") " Oct 31 22:11:39.005363 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.000445 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c8a256-3038-419a-b2df-fbf1991bfe30-openshift-service-ca" pod "17c8a256-3038-419a-b2df-fbf1991bfe30" (UID: "17c8a256-3038-419a-b2df-fbf1991bfe30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:39.024352 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.020353 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-metrics-certs" pod "17c8a256-3038-419a-b2df-fbf1991bfe30" (UID: "17c8a256-3038-419a-b2df-fbf1991bfe30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:11:39.073898 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.070876 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-maas-api-tls" pod "17c8a256-3038-419a-b2df-fbf1991bfe30" (UID: "17c8a256-3038-419a-b2df-fbf1991bfe30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:11:39.074095 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.073997 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c8a256-3038-419a-b2df-fbf1991bfe30-kube-api-access-mxmfd" pod "17c8a256-3038-419a-b2df-fbf1991bfe30" (UID: "17c8a256-3038-419a-b2df-fbf1991bfe30"). InnerVolumeSpecName "kube-api-access-mxmfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:39.074748 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.074714 5188 generic.go:356] "Generic (PLEG): container finished" podID="17c8a256-3038-419a-b2df-fbf1991bfe30" containerID="4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493" exitCode=0 Oct 31 22:11:39.074887 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.074849 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" event={"ID":"17c8a256-3038-419a-b2df-fbf1991bfe30","Type":"ContainerDied","Data":"4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493"} Oct 31 22:11:39.074959 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.074915 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" event={"ID":"17c8a256-3038-419a-b2df-fbf1991bfe30","Type":"ContainerDied","Data":"0b7dffd7ed720a02c624705026169e88e121e37baf6df9203d9db2f6eca03ffc"} Oct 31 22:11:39.074959 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.074954 5188 scope.go:122] "RemoveContainer" containerID="4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493" Oct 31 22:11:39.075405 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.075143 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5" Oct 31 22:11:39.088244 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.088153 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:39.088244 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.088184 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8a256-3038-419a-b2df-fbf1991bfe30-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:39.088244 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.088201 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8a256-3038-419a-b2df-fbf1991bfe30-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:39.088244 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.088216 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxmfd\" (UniqueName: \"kubernetes.io/projected/17c8a256-3038-419a-b2df-fbf1991bfe30-kube-api-access-mxmfd\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:39.102189 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.102149 5188 scope.go:122] "RemoveContainer" containerID="4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493" Oct 31 22:11:39.112901 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:11:39.112548 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493\": container with ID starting with 4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493 not found: ID does not exist" containerID="4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493" Oct 31 22:11:39.112901 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.112603 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493"} err="failed to get container status \"4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493\": rpc error: code = NotFound desc = could not find container \"4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493\": container with ID starting with 4a28466ae9bedae2648b62951f8b4deb99ef40d1e862791563e54d1fd9ec6493 not found: ID does not exist" Oct 31 22:11:39.139422 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.139382 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5"] Oct 31 22:11:39.142972 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.142934 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-5237cd-fb656c7db-p7gz5"] Oct 31 22:11:39.476073 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.476023 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c8a256-3038-419a-b2df-fbf1991bfe30" path="/var/lib/kubelet/pods/17c8a256-3038-419a-b2df-fbf1991bfe30/volumes" Oct 31 22:11:39.996253 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:39.996219 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:11:47.529743 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:47.529697 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" podUID="83480925-f78a-400c-83f5-6d44260c7d3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:11:47.530154 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:47.529747 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" podUID="6668ee01-98ae-4bbc-b134-3df50eeeb23f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:11:52.453676 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:52.453621 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4"] Oct 31 22:11:52.464017 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:52.463708 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m"] Oct 31 22:11:56.848066 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.848038 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 22:11:56.867437 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.867411 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 22:11:56.996458 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.996359 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83480925-f78a-400c-83f5-6d44260c7d3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83480925-f78a-400c-83f5-6d44260c7d3c-plugins-config-volume\") pod \"83480925-f78a-400c-83f5-6d44260c7d3c\" (UID: \"83480925-f78a-400c-83f5-6d44260c7d3c\") " Oct 31 22:11:56.996648 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.996472 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6668ee01-98ae-4bbc-b134-3df50eeeb23f-kube-api-access-kcphq\" (UniqueName: \"kubernetes.io/projected/6668ee01-98ae-4bbc-b134-3df50eeeb23f-kube-api-access-kcphq\") pod \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\" (UID: \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\") " Oct 31 22:11:56.996724 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.996646 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83480925-f78a-400c-83f5-6d44260c7d3c-kube-api-access-qmdpt\" (UniqueName: \"kubernetes.io/projected/83480925-f78a-400c-83f5-6d44260c7d3c-kube-api-access-qmdpt\") pod \"83480925-f78a-400c-83f5-6d44260c7d3c\" (UID: \"83480925-f78a-400c-83f5-6d44260c7d3c\") " Oct 31 22:11:56.996724 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.996666 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6668ee01-98ae-4bbc-b134-3df50eeeb23f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6668ee01-98ae-4bbc-b134-3df50eeeb23f-plugins-config-volume\") pod \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\" (UID: \"6668ee01-98ae-4bbc-b134-3df50eeeb23f\") " Oct 31 22:11:56.996839 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.996779 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83480925-f78a-400c-83f5-6d44260c7d3c-plugins-config-volume" pod "83480925-f78a-400c-83f5-6d44260c7d3c" (UID: "83480925-f78a-400c-83f5-6d44260c7d3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:56.997216 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.997194 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6668ee01-98ae-4bbc-b134-3df50eeeb23f-plugins-config-volume" pod "6668ee01-98ae-4bbc-b134-3df50eeeb23f" (UID: "6668ee01-98ae-4bbc-b134-3df50eeeb23f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:56.997394 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.997375 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6668ee01-98ae-4bbc-b134-3df50eeeb23f-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:56.997474 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.997396 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83480925-f78a-400c-83f5-6d44260c7d3c-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:56.999099 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.999076 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83480925-f78a-400c-83f5-6d44260c7d3c-kube-api-access-qmdpt" pod "83480925-f78a-400c-83f5-6d44260c7d3c" (UID: "83480925-f78a-400c-83f5-6d44260c7d3c"). InnerVolumeSpecName "kube-api-access-qmdpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:56.999249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:56.999211 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6668ee01-98ae-4bbc-b134-3df50eeeb23f-kube-api-access-kcphq" pod "6668ee01-98ae-4bbc-b134-3df50eeeb23f" (UID: "6668ee01-98ae-4bbc-b134-3df50eeeb23f"). InnerVolumeSpecName "kube-api-access-kcphq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:57.071623 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.071586 5188 scope.go:122] "RemoveContainer" containerID="463ded2a1846fde73af96e4f4732ba44d3f7ecff84e0ddee5d5f9e6946489c80" Oct 31 22:11:57.086940 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.086611 5188 scope.go:122] "RemoveContainer" containerID="a8fc7227f12f63e601b188183b1c75936ead110da58ecd6611171362a6395084" Oct 31 22:11:57.098220 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.098192 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmdpt\" (UniqueName: \"kubernetes.io/projected/83480925-f78a-400c-83f5-6d44260c7d3c-kube-api-access-qmdpt\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:57.098335 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.098223 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcphq\" (UniqueName: \"kubernetes.io/projected/6668ee01-98ae-4bbc-b134-3df50eeeb23f-kube-api-access-kcphq\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:11:57.268977 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.268875 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" event={"ID":"83480925-f78a-400c-83f5-6d44260c7d3c","Type":"ContainerDied","Data":"a8fc7227f12f63e601b188183b1c75936ead110da58ecd6611171362a6395084"} Oct 31 22:11:57.268977 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.268946 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" event={"ID":"83480925-f78a-400c-83f5-6d44260c7d3c","Type":"ContainerDied","Data":"c0029698b8071ff799b6137d23f639069cd8ea87db0716a3c97aabcbb87c57e6"} Oct 31 22:11:57.268977 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.268974 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" event={"ID":"6668ee01-98ae-4bbc-b134-3df50eeeb23f","Type":"ContainerDied","Data":"463ded2a1846fde73af96e4f4732ba44d3f7ecff84e0ddee5d5f9e6946489c80"} Oct 31 22:11:57.269270 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.268980 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" Oct 31 22:11:57.269270 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.268996 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n" event={"ID":"6668ee01-98ae-4bbc-b134-3df50eeeb23f","Type":"ContainerDied","Data":"03b0c6c7a143138d66ee04f6aab6e71d0bf7ae95dc37188a123a37a40375ee02"} Oct 31 22:11:57.269270 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.268980 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz" Oct 31 22:11:57.373358 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.373285 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n"] Oct 31 22:11:57.379838 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.379803 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-5237cd-57bcf4f5cf-5fw8n"] Oct 31 22:11:57.409963 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.409919 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz"] Oct 31 22:11:57.417264 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.417231 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-5237cd-657746898c-q7drz"] Oct 31 22:11:57.476812 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.476773 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6668ee01-98ae-4bbc-b134-3df50eeeb23f" path="/var/lib/kubelet/pods/6668ee01-98ae-4bbc-b134-3df50eeeb23f/volumes" Oct 31 22:11:57.477486 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:57.477464 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83480925-f78a-400c-83f5-6d44260c7d3c" path="/var/lib/kubelet/pods/83480925-f78a-400c-83f5-6d44260c7d3c/volumes" Oct 31 22:11:58.215935 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:11:58.215878 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b"] Oct 31 22:12:01.603548 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.603497 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm"] Oct 31 22:12:01.604614 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.604586 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6668ee01-98ae-4bbc-b134-3df50eeeb23f" containerName="payload-processing" Oct 31 22:12:01.604614 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.604614 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6668ee01-98ae-4bbc-b134-3df50eeeb23f" containerName="payload-processing" Oct 31 22:12:01.604835 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.604634 5188 cpu_manager.go:414] "containerMap: removing container" podUID="17c8a256-3038-419a-b2df-fbf1991bfe30" containerName="maas-api" Oct 31 22:12:01.604835 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.604642 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c8a256-3038-419a-b2df-fbf1991bfe30" containerName="maas-api" Oct 31 22:12:01.604835 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.604679 5188 cpu_manager.go:414] "containerMap: removing container" podUID="83480925-f78a-400c-83f5-6d44260c7d3c" containerName="payload-pre-processing" Oct 31 22:12:01.604835 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.604690 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83480925-f78a-400c-83f5-6d44260c7d3c" containerName="payload-pre-processing" Oct 31 22:12:01.605048 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.604909 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6668ee01-98ae-4bbc-b134-3df50eeeb23f" containerName="payload-processing" Oct 31 22:12:01.605048 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.604920 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c8a256-3038-419a-b2df-fbf1991bfe30" containerName="maas-api" Oct 31 22:12:01.605048 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.604938 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="83480925-f78a-400c-83f5-6d44260c7d3c" containerName="payload-pre-processing" Oct 31 22:12:01.610846 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.610819 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.625583 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.625552 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1b0cce-serving-cert\"" Oct 31 22:12:01.627781 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.627412 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm"] Oct 31 22:12:01.679120 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.679085 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-metrics-certs\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.679367 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.679134 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.679367 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.679242 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0063-d7e6-4434-861f-051bef5e818f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.679547 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.679397 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhwf\" (UniqueName: \"kubernetes.io/projected/906b0063-d7e6-4434-861f-051bef5e818f-kube-api-access-glhwf\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.780650 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.780595 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhwf\" (UniqueName: \"kubernetes.io/projected/906b0063-d7e6-4434-861f-051bef5e818f-kube-api-access-glhwf\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.780871 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.780734 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-metrics-certs\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.780871 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.780774 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.780871 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.780829 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0063-d7e6-4434-861f-051bef5e818f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.781732 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.781700 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0063-d7e6-4434-861f-051bef5e818f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.783727 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.783695 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-metrics-certs\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.784119 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.784097 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.814624 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.814530 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhwf\" (UniqueName: \"kubernetes.io/projected/906b0063-d7e6-4434-861f-051bef5e818f-kube-api-access-glhwf\") pod \"maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:01.937759 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:01.937722 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:02.262844 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:02.262816 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm"] Oct 31 22:12:02.263663 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:12:02.263630 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906b0063_d7e6_4434_861f_051bef5e818f.slice/crio-5713b4be864b70b3a01d67e5e3f4b17dfa7f651d5424c31679aa542b0fe4dd95 WatchSource:0}: Error finding container 5713b4be864b70b3a01d67e5e3f4b17dfa7f651d5424c31679aa542b0fe4dd95: Status 404 returned error can't find the container with id 5713b4be864b70b3a01d67e5e3f4b17dfa7f651d5424c31679aa542b0fe4dd95 Oct 31 22:12:02.317558 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:02.317515 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" event={"ID":"906b0063-d7e6-4434-861f-051bef5e818f","Type":"ContainerStarted","Data":"5713b4be864b70b3a01d67e5e3f4b17dfa7f651d5424c31679aa542b0fe4dd95"} Oct 31 22:12:02.541528 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:12:02.541435 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:12:03.216570 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:03.216522 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" podUID="4433702b-b648-49e8-9aa5-12bcdc71b275" containerName="payload-processing" containerID="cri-o://0bb1d391c9765c248113dd339f208296a9a7987c99edc7f6f6439816c1172415" gracePeriod=25 Oct 31 22:12:03.327694 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:03.327647 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" event={"ID":"906b0063-d7e6-4434-861f-051bef5e818f","Type":"ContainerStarted","Data":"dc1a25d8c3a1a9aa0ce8aa2c2752456a3e1c2a9c41799b0a9fa8d6f126f983ab"} Oct 31 22:12:03.327899 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:03.327729 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:03.367667 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:03.367608 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" podStartSLOduration=2.063293155 podStartE2EDuration="2.367591988s" totalImagesPullingTime="304.298833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:12:03.364685682 +0000 UTC m=+3370.057643088" watchObservedRunningTime="2026-10-31 22:12:03.367591988 +0000 UTC m=+3370.060549380" Oct 31 22:12:07.486596 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.486497 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" containerID="cri-o://c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771" gracePeriod=45 Oct 31 22:12:07.492751 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.492679 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" containerID="cri-o://cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38" gracePeriod=45 Oct 31 22:12:07.795587 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.795519 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" podUID="4433702b-b648-49e8-9aa5-12bcdc71b275" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:12:07.799665 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.799636 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 22:12:07.827693 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.827653 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 22:12:07.896706 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.896665 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tmp-dir\") pod \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " Oct 31 22:12:07.896897 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.896750 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-tmp-dir\") pod \"c1176d9d-4701-4c7d-84ec-60f664b45470\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " Oct 31 22:12:07.896897 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.896799 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-dshm\") pod \"c1176d9d-4701-4c7d-84ec-60f664b45470\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " Oct 31 22:12:07.896897 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.896846 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1176d9d-4701-4c7d-84ec-60f664b45470-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1176d9d-4701-4c7d-84ec-60f664b45470-tls-certs\") pod \"c1176d9d-4701-4c7d-84ec-60f664b45470\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " Oct 31 22:12:07.897017 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.896903 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-kube-api-access-xwrgk\" (UniqueName: \"kubernetes.io/projected/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-kube-api-access-xwrgk\") pod \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " Oct 31 22:12:07.897017 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.896958 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-home\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-home\") pod \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " Oct 31 22:12:07.897017 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.897006 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-dshm\") pod \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " Oct 31 22:12:07.897118 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.897075 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-home\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-home\") pod \"c1176d9d-4701-4c7d-84ec-60f664b45470\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " Oct 31 22:12:07.897157 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.897122 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tls-certs\") pod \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " Oct 31 22:12:07.897193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.897165 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1176d9d-4701-4c7d-84ec-60f664b45470-kube-api-access-l74pc\" (UniqueName: \"kubernetes.io/projected/c1176d9d-4701-4c7d-84ec-60f664b45470-kube-api-access-l74pc\") pod \"c1176d9d-4701-4c7d-84ec-60f664b45470\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " Oct 31 22:12:07.897261 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.897242 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-model-cache\") pod \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\" (UID: \"6b18e0f7-04d4-4854-b47b-27b07dde1b1b\") " Oct 31 22:12:07.897337 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.897303 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-model-cache\") pod \"c1176d9d-4701-4c7d-84ec-60f664b45470\" (UID: \"c1176d9d-4701-4c7d-84ec-60f664b45470\") " Oct 31 22:12:07.900050 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.898897 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-home" pod "6b18e0f7-04d4-4854-b47b-27b07dde1b1b" (UID: "6b18e0f7-04d4-4854-b47b-27b07dde1b1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.900050 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.898941 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tmp-dir" pod "6b18e0f7-04d4-4854-b47b-27b07dde1b1b" (UID: "6b18e0f7-04d4-4854-b47b-27b07dde1b1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.900050 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.899262 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-model-cache" pod "c1176d9d-4701-4c7d-84ec-60f664b45470" (UID: "c1176d9d-4701-4c7d-84ec-60f664b45470"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.900050 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.899718 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-model-cache" pod "6b18e0f7-04d4-4854-b47b-27b07dde1b1b" (UID: "6b18e0f7-04d4-4854-b47b-27b07dde1b1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.900466 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.900192 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-home" pod "c1176d9d-4701-4c7d-84ec-60f664b45470" (UID: "c1176d9d-4701-4c7d-84ec-60f664b45470"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.901485 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.901447 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-tmp-dir" pod "c1176d9d-4701-4c7d-84ec-60f664b45470" (UID: "c1176d9d-4701-4c7d-84ec-60f664b45470"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.903051 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.903001 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-dshm" pod "c1176d9d-4701-4c7d-84ec-60f664b45470" (UID: "c1176d9d-4701-4c7d-84ec-60f664b45470"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.903870 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.903835 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1176d9d-4701-4c7d-84ec-60f664b45470-kube-api-access-l74pc" pod "c1176d9d-4701-4c7d-84ec-60f664b45470" (UID: "c1176d9d-4701-4c7d-84ec-60f664b45470"). InnerVolumeSpecName "kube-api-access-l74pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:12:07.904745 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.904714 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1176d9d-4701-4c7d-84ec-60f664b45470-tls-certs" pod "c1176d9d-4701-4c7d-84ec-60f664b45470" (UID: "c1176d9d-4701-4c7d-84ec-60f664b45470"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:12:07.904870 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.904825 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tls-certs" pod "6b18e0f7-04d4-4854-b47b-27b07dde1b1b" (UID: "6b18e0f7-04d4-4854-b47b-27b07dde1b1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:12:07.905728 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.905698 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-kube-api-access-xwrgk" pod "6b18e0f7-04d4-4854-b47b-27b07dde1b1b" (UID: "6b18e0f7-04d4-4854-b47b-27b07dde1b1b"). InnerVolumeSpecName "kube-api-access-xwrgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:12:07.907194 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:07.906981 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-dshm" pod "6b18e0f7-04d4-4854-b47b-27b07dde1b1b" (UID: "6b18e0f7-04d4-4854-b47b-27b07dde1b1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:08.000961 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.000908 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.000961 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.000962 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.000977 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1176d9d-4701-4c7d-84ec-60f664b45470-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.000993 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrgk\" (UniqueName: \"kubernetes.io/projected/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-kube-api-access-xwrgk\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.001007 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.001019 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.001030 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.001042 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.001055 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74pc\" (UniqueName: \"kubernetes.io/projected/c1176d9d-4701-4c7d-84ec-60f664b45470-kube-api-access-l74pc\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.001069 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.001082 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1176d9d-4701-4c7d-84ec-60f664b45470-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.001249 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.001095 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b18e0f7-04d4-4854-b47b-27b07dde1b1b-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.371603 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.371562 5188 generic.go:356] "Generic (PLEG): container finished" podID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerID="c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771" exitCode=0 Oct 31 22:12:08.371837 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.371631 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" Oct 31 22:12:08.371837 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.371647 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" event={"ID":"c1176d9d-4701-4c7d-84ec-60f664b45470","Type":"ContainerDied","Data":"c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771"} Oct 31 22:12:08.371837 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.371694 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4" event={"ID":"c1176d9d-4701-4c7d-84ec-60f664b45470","Type":"ContainerDied","Data":"6c0c009e7c329ccf444121d651ec71205d1d4396c681e9c9466edbf2874accdd"} Oct 31 22:12:08.371837 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.371718 5188 scope.go:122] "RemoveContainer" containerID="c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771" Oct 31 22:12:08.373761 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.373732 5188 generic.go:356] "Generic (PLEG): container finished" podID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerID="cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38" exitCode=0 Oct 31 22:12:08.373889 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.373783 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" event={"ID":"6b18e0f7-04d4-4854-b47b-27b07dde1b1b","Type":"ContainerDied","Data":"cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38"} Oct 31 22:12:08.373889 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.373817 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" Oct 31 22:12:08.373889 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.373841 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m" event={"ID":"6b18e0f7-04d4-4854-b47b-27b07dde1b1b","Type":"ContainerDied","Data":"d87f2527eb1dc15fb28bc894f6dd4002366ecd68f76a17bd0d0051bf1fe417db"} Oct 31 22:12:08.383629 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.383604 5188 scope.go:122] "RemoveContainer" containerID="c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771" Oct 31 22:12:08.383974 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:12:08.383951 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771\": container with ID starting with c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771 not found: ID does not exist" containerID="c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771" Oct 31 22:12:08.384067 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.383986 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771"} err="failed to get container status \"c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771\": rpc error: code = NotFound desc = could not find container \"c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771\": container with ID starting with c6147ca641a533c513bc3e404f889973ac62e8543fcd735cb5ed075ab5591771 not found: ID does not exist" Oct 31 22:12:08.384067 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.384014 5188 scope.go:122] "RemoveContainer" containerID="cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38" Oct 31 22:12:08.396118 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.396094 5188 scope.go:122] "RemoveContainer" containerID="cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38" Oct 31 22:12:08.396490 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:12:08.396464 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38\": container with ID starting with cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38 not found: ID does not exist" containerID="cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38" Oct 31 22:12:08.396586 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.396503 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38"} err="failed to get container status \"cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38\": rpc error: code = NotFound desc = could not find container \"cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38\": container with ID starting with cf7e668024f6edcf57e89c7cbea3aa3f36450d40770ca898c4dc960aba8e2b38 not found: ID does not exist" Oct 31 22:12:08.424059 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.424021 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m"] Oct 31 22:12:08.435631 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.435597 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/simulator-w0-4b5989-kserve-ddf7b6758-7ld8m"] Oct 31 22:12:08.457686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.457646 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4"] Oct 31 22:12:08.475546 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:08.475500 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/premium-w0-4b5989-kserve-dd687bf47-qtwv4"] Oct 31 22:12:09.341995 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:09.341962 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:12:09.474636 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:09.474599 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" path="/var/lib/kubelet/pods/6b18e0f7-04d4-4854-b47b-27b07dde1b1b/volumes" Oct 31 22:12:09.475097 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:09.475082 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" path="/var/lib/kubelet/pods/c1176d9d-4701-4c7d-84ec-60f664b45470/volumes" Oct 31 22:12:17.796263 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:17.796219 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" podUID="4433702b-b648-49e8-9aa5-12bcdc71b275" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:12:23.494994 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.494892 5188 generic.go:356] "Generic (PLEG): container finished" podID="4433702b-b648-49e8-9aa5-12bcdc71b275" containerID="0bb1d391c9765c248113dd339f208296a9a7987c99edc7f6f6439816c1172415" exitCode=0 Oct 31 22:12:23.494994 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.494966 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" event={"ID":"4433702b-b648-49e8-9aa5-12bcdc71b275","Type":"ContainerDied","Data":"0bb1d391c9765c248113dd339f208296a9a7987c99edc7f6f6439816c1172415"} Oct 31 22:12:23.527000 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.526975 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 22:12:23.652965 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.652932 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4433702b-b648-49e8-9aa5-12bcdc71b275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4433702b-b648-49e8-9aa5-12bcdc71b275-plugins-config-volume\") pod \"4433702b-b648-49e8-9aa5-12bcdc71b275\" (UID: \"4433702b-b648-49e8-9aa5-12bcdc71b275\") " Oct 31 22:12:23.653193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.653057 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4433702b-b648-49e8-9aa5-12bcdc71b275-kube-api-access-cbqhb\" (UniqueName: \"kubernetes.io/projected/4433702b-b648-49e8-9aa5-12bcdc71b275-kube-api-access-cbqhb\") pod \"4433702b-b648-49e8-9aa5-12bcdc71b275\" (UID: \"4433702b-b648-49e8-9aa5-12bcdc71b275\") " Oct 31 22:12:23.653382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.653356 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4433702b-b648-49e8-9aa5-12bcdc71b275-plugins-config-volume" pod "4433702b-b648-49e8-9aa5-12bcdc71b275" (UID: "4433702b-b648-49e8-9aa5-12bcdc71b275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:12:23.653687 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.653669 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4433702b-b648-49e8-9aa5-12bcdc71b275-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:23.655578 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.655540 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4433702b-b648-49e8-9aa5-12bcdc71b275-kube-api-access-cbqhb" pod "4433702b-b648-49e8-9aa5-12bcdc71b275" (UID: "4433702b-b648-49e8-9aa5-12bcdc71b275"). InnerVolumeSpecName "kube-api-access-cbqhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:12:23.755241 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.755154 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbqhb\" (UniqueName: \"kubernetes.io/projected/4433702b-b648-49e8-9aa5-12bcdc71b275-kube-api-access-cbqhb\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:12:23.766986 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.766941 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2"] Oct 31 22:12:23.767992 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.767961 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" Oct 31 22:12:23.767992 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.767989 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" Oct 31 22:12:23.768221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.768062 5188 cpu_manager.go:414] "containerMap: removing container" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" Oct 31 22:12:23.768221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.768071 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" Oct 31 22:12:23.768221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.768103 5188 cpu_manager.go:414] "containerMap: removing container" podUID="4433702b-b648-49e8-9aa5-12bcdc71b275" containerName="payload-processing" Oct 31 22:12:23.768221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.768111 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4433702b-b648-49e8-9aa5-12bcdc71b275" containerName="payload-processing" Oct 31 22:12:23.768500 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.768274 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1176d9d-4701-4c7d-84ec-60f664b45470" containerName="main" Oct 31 22:12:23.768500 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.768292 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="4433702b-b648-49e8-9aa5-12bcdc71b275" containerName="payload-processing" Oct 31 22:12:23.768500 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.768307 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b18e0f7-04d4-4854-b47b-27b07dde1b1b" containerName="main" Oct 31 22:12:23.772316 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.772292 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.805678 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.805634 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cd421c\"/\"test-model-cd421c-kserve-self-signed-certs\"" Oct 31 22:12:23.805871 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.805677 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cd421c\"/\"openshift-service-ca.crt\"" Oct 31 22:12:23.805871 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.805705 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cd421c\"/\"kube-root-ca.crt\"" Oct 31 22:12:23.805871 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.805826 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cd421c\"/\"default-dockercfg-tl9np\"" Oct 31 22:12:23.810781 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.810749 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2"] Oct 31 22:12:23.856239 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.856201 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-tmp-dir\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.856451 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.856306 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-model-cache\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.856451 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.856413 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-dshm\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.856451 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.856446 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbx2s\" (UniqueName: \"kubernetes.io/projected/2cb608c4-b046-4592-b919-6a26f69f6828-kube-api-access-bbx2s\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.856629 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.856590 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb608c4-b046-4592-b919-6a26f69f6828-tls-certs\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.856742 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.856725 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-home\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.957538 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.957502 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-home\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.957757 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.957573 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-tmp-dir\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.957757 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.957650 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-model-cache\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.957757 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.957719 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-dshm\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.957952 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.957765 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbx2s\" (UniqueName: \"kubernetes.io/projected/2cb608c4-b046-4592-b919-6a26f69f6828-kube-api-access-bbx2s\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.957952 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.957853 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb608c4-b046-4592-b919-6a26f69f6828-tls-certs\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.958076 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.957968 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-home\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.958076 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.958034 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-tmp-dir\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.958304 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.958286 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-model-cache\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.960127 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.960101 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-dshm\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.960494 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.960472 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb608c4-b046-4592-b919-6a26f69f6828-tls-certs\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:23.981689 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:23.981651 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbx2s\" (UniqueName: \"kubernetes.io/projected/2cb608c4-b046-4592-b919-6a26f69f6828-kube-api-access-bbx2s\") pod \"test-model-cd421c-kserve-65f97f47f7-4f5q2\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:24.121757 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:24.121448 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:24.451958 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:24.451857 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2"] Oct 31 22:12:24.456385 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:12:24.456308 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb608c4_b046_4592_b919_6a26f69f6828.slice/crio-d0369dd473448714e1207345099b214ef9286376b7ad96d6fa6c1820b349e9d5 WatchSource:0}: Error finding container d0369dd473448714e1207345099b214ef9286376b7ad96d6fa6c1820b349e9d5: Status 404 returned error can't find the container with id d0369dd473448714e1207345099b214ef9286376b7ad96d6fa6c1820b349e9d5 Oct 31 22:12:24.503663 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:24.503621 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" event={"ID":"2cb608c4-b046-4592-b919-6a26f69f6828","Type":"ContainerStarted","Data":"d0369dd473448714e1207345099b214ef9286376b7ad96d6fa6c1820b349e9d5"} Oct 31 22:12:24.505542 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:24.505506 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" event={"ID":"4433702b-b648-49e8-9aa5-12bcdc71b275","Type":"ContainerDied","Data":"565d309e985ce3bd851166f324fb5c6010eebca9aa1ef384b9b0f8666366a1af"} Oct 31 22:12:24.505686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:24.505582 5188 scope.go:122] "RemoveContainer" containerID="0bb1d391c9765c248113dd339f208296a9a7987c99edc7f6f6439816c1172415" Oct 31 22:12:24.505686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:24.505520 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b" Oct 31 22:12:24.551686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:24.551649 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b"] Oct 31 22:12:24.556387 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:24.556362 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-4b5989-94b469d6f-qd76b"] Oct 31 22:12:25.475438 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:25.475395 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4433702b-b648-49e8-9aa5-12bcdc71b275" path="/var/lib/kubelet/pods/4433702b-b648-49e8-9aa5-12bcdc71b275/volumes" Oct 31 22:12:25.519213 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:25.519168 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" event={"ID":"2cb608c4-b046-4592-b919-6a26f69f6828","Type":"ContainerStarted","Data":"6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6"} Oct 31 22:12:25.558786 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:25.558719 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" podStartSLOduration=2.5587026059999998 podStartE2EDuration="2.558702606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:12:25.556091763 +0000 UTC m=+3392.249049155" watchObservedRunningTime="2026-10-31 22:12:25.558702606 +0000 UTC m=+3392.251659998" Oct 31 22:12:26.121889 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:26.121842 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:32.543473 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:12:32.543431 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:12:34.122260 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:34.122194 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:34.140199 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:34.140159 5188 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:34.607266 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:34.607179 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:12:54.537221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.537182 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq"] Oct 31 22:12:54.648636 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.648590 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq"] Oct 31 22:12:54.648859 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.648724 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:12:54.665149 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.664969 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5c54fc-dockercfg-scq4b\"" Oct 31 22:12:54.674930 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.674891 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5c54fc\"" Oct 31 22:12:54.762226 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.762185 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq\" (UID: \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:12:54.762519 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.762488 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65mnq\" (UniqueName: \"kubernetes.io/projected/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-kube-api-access-65mnq\") pod \"payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq\" (UID: \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:12:54.863795 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.863754 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65mnq\" (UniqueName: \"kubernetes.io/projected/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-kube-api-access-65mnq\") pod \"payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq\" (UID: \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:12:54.863988 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.863897 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq\" (UID: \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:12:54.864691 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.864660 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq\" (UID: \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:12:54.885344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.885292 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65mnq\" (UniqueName: \"kubernetes.io/projected/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-kube-api-access-65mnq\") pod \"payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq\" (UID: \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:12:54.983618 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:54.983574 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:12:55.327218 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:55.327182 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq"] Oct 31 22:12:55.330831 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:12:55.330796 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ab05c6_b656_41d0_982b_3c5b0e8e07c4.slice/crio-282cec757327e8bf0619efea14455ecab47db07bc4721f002662a461c2a7976b WatchSource:0}: Error finding container 282cec757327e8bf0619efea14455ecab47db07bc4721f002662a461c2a7976b: Status 404 returned error can't find the container with id 282cec757327e8bf0619efea14455ecab47db07bc4721f002662a461c2a7976b Oct 31 22:12:55.806298 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:55.806253 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" event={"ID":"52ab05c6-b656-41d0-982b-3c5b0e8e07c4","Type":"ContainerStarted","Data":"c27edaf9cf97225aff1dc7358922ea37d6bfc26edf3c45e1910e565fd5aafd94"} Oct 31 22:12:55.806863 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:55.806310 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" event={"ID":"52ab05c6-b656-41d0-982b-3c5b0e8e07c4","Type":"ContainerStarted","Data":"282cec757327e8bf0619efea14455ecab47db07bc4721f002662a461c2a7976b"} Oct 31 22:12:55.806863 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:55.806580 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:12:55.845614 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:12:55.845549 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" podStartSLOduration=1.845532586 podStartE2EDuration="1.845532586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:12:55.842340212 +0000 UTC m=+3422.535297598" watchObservedRunningTime="2026-10-31 22:12:55.845532586 +0000 UTC m=+3422.538489990" Oct 31 22:13:02.539733 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:13:02.539691 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:13:06.817784 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:06.817748 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:13:16.566176 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:16.566131 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k"] Oct 31 22:13:16.566727 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:16.566434 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" containerID="cri-o://ecd68dde986329c1b4fa0abb255df0e0d7c69b92e7ce896e232388b6d38f998e" gracePeriod=30 Oct 31 22:13:21.390052 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:21.390002 5188 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:13:21.390592 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:21.390082 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:13:22.088186 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.088139 5188 generic.go:356] "Generic (PLEG): container finished" podID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerID="ecd68dde986329c1b4fa0abb255df0e0d7c69b92e7ce896e232388b6d38f998e" exitCode=0 Oct 31 22:13:22.088390 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.088217 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" event={"ID":"f27c03e2-52d5-48b7-89c9-36bc18aa09db","Type":"ContainerDied","Data":"ecd68dde986329c1b4fa0abb255df0e0d7c69b92e7ce896e232388b6d38f998e"} Oct 31 22:13:22.272889 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.272858 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 22:13:22.407086 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407043 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-podinfo\") pod \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " Oct 31 22:13:22.407686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407140 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-kube-api-access-xcxdb\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-kube-api-access-xcxdb\") pod \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " Oct 31 22:13:22.407686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407176 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-credential-socket\") pod \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " Oct 31 22:13:22.407686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407227 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istiod-ca-cert\") pod \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " Oct 31 22:13:22.407686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407372 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-certs\") pod \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " Oct 31 22:13:22.407686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407410 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-data\") pod \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " Oct 31 22:13:22.407686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407515 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-token\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-token\") pod \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " Oct 31 22:13:22.407686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407534 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-credential-socket" pod "f27c03e2-52d5-48b7-89c9-36bc18aa09db" (UID: "f27c03e2-52d5-48b7-89c9-36bc18aa09db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:22.407686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407575 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-envoy\") pod \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " Oct 31 22:13:22.408159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407689 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-socket\") pod \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\" (UID: \"f27c03e2-52d5-48b7-89c9-36bc18aa09db\") " Oct 31 22:13:22.408159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.407826 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-certs" pod "f27c03e2-52d5-48b7-89c9-36bc18aa09db" (UID: "f27c03e2-52d5-48b7-89c9-36bc18aa09db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:22.408159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.408104 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-data" pod "f27c03e2-52d5-48b7-89c9-36bc18aa09db" (UID: "f27c03e2-52d5-48b7-89c9-36bc18aa09db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:22.408159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.408118 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-socket" pod "f27c03e2-52d5-48b7-89c9-36bc18aa09db" (UID: "f27c03e2-52d5-48b7-89c9-36bc18aa09db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:22.408446 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.408424 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.408513 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.408448 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.408513 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.408462 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.408513 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.408476 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.408676 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.408658 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istiod-ca-cert" pod "f27c03e2-52d5-48b7-89c9-36bc18aa09db" (UID: "f27c03e2-52d5-48b7-89c9-36bc18aa09db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:13:22.410159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.410121 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-envoy" pod "f27c03e2-52d5-48b7-89c9-36bc18aa09db" (UID: "f27c03e2-52d5-48b7-89c9-36bc18aa09db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:22.410159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.410127 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-kube-api-access-xcxdb" pod "f27c03e2-52d5-48b7-89c9-36bc18aa09db" (UID: "f27c03e2-52d5-48b7-89c9-36bc18aa09db"). InnerVolumeSpecName "kube-api-access-xcxdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:22.410390 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.410190 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-podinfo" pod "f27c03e2-52d5-48b7-89c9-36bc18aa09db" (UID: "f27c03e2-52d5-48b7-89c9-36bc18aa09db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:13:22.410655 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.410619 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-token" pod "f27c03e2-52d5-48b7-89c9-36bc18aa09db" (UID: "f27c03e2-52d5-48b7-89c9-36bc18aa09db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:22.509946 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.509905 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.509946 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.509944 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.510181 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.509957 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcxdb\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-kube-api-access-xcxdb\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.510181 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.509973 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.510181 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:22.509984 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f27c03e2-52d5-48b7-89c9-36bc18aa09db-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:13:23.098724 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:23.098684 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" Oct 31 22:13:23.098724 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:23.098702 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k" event={"ID":"f27c03e2-52d5-48b7-89c9-36bc18aa09db","Type":"ContainerDied","Data":"42ed3aceee3e867aefd8d5dd7db78eafa5bdd2805a36b938979beb1fa5fdaaa4"} Oct 31 22:13:23.099279 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:23.098785 5188 scope.go:122] "RemoveContainer" containerID="ecd68dde986329c1b4fa0abb255df0e0d7c69b92e7ce896e232388b6d38f998e" Oct 31 22:13:23.160232 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:23.160179 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k"] Oct 31 22:13:23.169835 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:23.169793 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-hvq2k"] Oct 31 22:13:23.474638 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:23.474595 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" path="/var/lib/kubelet/pods/f27c03e2-52d5-48b7-89c9-36bc18aa09db/volumes" Oct 31 22:13:32.540605 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:13:32.540560 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:13:32.688417 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.688379 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk"] Oct 31 22:13:32.689279 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.689244 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" Oct 31 22:13:32.689279 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.689274 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" Oct 31 22:13:32.689550 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.689476 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27c03e2-52d5-48b7-89c9-36bc18aa09db" containerName="istio-proxy" Oct 31 22:13:32.693184 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.693161 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.705521 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.705493 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5c54fc-serving-cert\"" Oct 31 22:13:32.721226 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.721190 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk"] Oct 31 22:13:32.746844 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.746789 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.747034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.746854 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-metrics-certs\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.747134 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.747064 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.747227 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.747202 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbjq\" (UniqueName: \"kubernetes.io/projected/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-kube-api-access-6vbjq\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.848697 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.848598 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.848913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.848699 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbjq\" (UniqueName: \"kubernetes.io/projected/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-kube-api-access-6vbjq\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.848913 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:13:32.848770 5188 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-serving-cert: secret "maas-api-e2e-inf-b-5c54fc-serving-cert" not found Oct 31 22:13:32.848913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.848794 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.848913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.848821 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-metrics-certs\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.848913 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:13:32.848851 5188 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls podName:f353bffb-dcb8-43c9-b3ee-dd33bd811d0c nodeName:}" failed. No retries permitted until 2026-10-31 22:13:33.348831376 +0000 UTC m=+3460.041788747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls") pod "maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" (UID: "f353bffb-dcb8-43c9-b3ee-dd33bd811d0c") : secret "maas-api-e2e-inf-b-5c54fc-serving-cert" not found Oct 31 22:13:32.849618 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.849597 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.851405 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.851384 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-metrics-certs\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:32.920969 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:32.920229 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbjq\" (UniqueName: \"kubernetes.io/projected/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-kube-api-access-6vbjq\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:33.355562 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:33.355507 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:33.358234 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:33.358205 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls\") pod \"maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:33.613283 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:33.613247 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:33.929159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:33.929122 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk"] Oct 31 22:13:33.929924 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:13:33.929894 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf353bffb_dcb8_43c9_b3ee_dd33bd811d0c.slice/crio-db2b4ec7dc26322b0c57b9f2bdb4fb9243bd490fa45840733a5016705784fbde WatchSource:0}: Error finding container db2b4ec7dc26322b0c57b9f2bdb4fb9243bd490fa45840733a5016705784fbde: Status 404 returned error can't find the container with id db2b4ec7dc26322b0c57b9f2bdb4fb9243bd490fa45840733a5016705784fbde Oct 31 22:13:33.931456 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:33.931433 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 22:13:34.192150 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:34.192047 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" event={"ID":"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c","Type":"ContainerStarted","Data":"db2b4ec7dc26322b0c57b9f2bdb4fb9243bd490fa45840733a5016705784fbde"} Oct 31 22:13:35.201480 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:35.201438 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" event={"ID":"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c","Type":"ContainerStarted","Data":"2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669"} Oct 31 22:13:35.201982 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:35.201529 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:13:35.242974 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:35.242905 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" podStartSLOduration=2.819328223 podStartE2EDuration="3.242889438s" totalImagesPullingTime="423.561215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:13:35.23879978 +0000 UTC m=+3461.931757175" watchObservedRunningTime="2026-10-31 22:13:35.242889438 +0000 UTC m=+3461.935846854" Oct 31 22:13:41.216273 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:13:41.216236 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:14:02.539642 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:14:02.539595 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:14:27.578645 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:27.578587 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf"] Oct 31 22:14:27.581242 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:27.579739 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" podUID="da10e874-0602-4ca3-86aa-500bb5b32448" containerName="maas-api" containerID="cri-o://24491d40dc0d767cfa5e32d8e8b738d439858dea3949dff2cb4008e6c3db85fc" gracePeriod=30 Oct 31 22:14:27.740221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:27.740006 5188 generic.go:356] "Generic (PLEG): container finished" podID="da10e874-0602-4ca3-86aa-500bb5b32448" containerID="24491d40dc0d767cfa5e32d8e8b738d439858dea3949dff2cb4008e6c3db85fc" exitCode=0 Oct 31 22:14:27.740221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:27.740167 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" event={"ID":"da10e874-0602-4ca3-86aa-500bb5b32448","Type":"ContainerDied","Data":"24491d40dc0d767cfa5e32d8e8b738d439858dea3949dff2cb4008e6c3db85fc"} Oct 31 22:14:27.964125 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:27.964094 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 22:14:28.143113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.143006 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-maas-api-tls\") pod \"da10e874-0602-4ca3-86aa-500bb5b32448\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " Oct 31 22:14:28.143489 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.143444 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da10e874-0602-4ca3-86aa-500bb5b32448-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da10e874-0602-4ca3-86aa-500bb5b32448-openshift-service-ca\") pod \"da10e874-0602-4ca3-86aa-500bb5b32448\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " Oct 31 22:14:28.143873 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.143854 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-metrics-certs\") pod \"da10e874-0602-4ca3-86aa-500bb5b32448\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " Oct 31 22:14:28.144313 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.144290 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da10e874-0602-4ca3-86aa-500bb5b32448-kube-api-access-xdd6q\" (UniqueName: \"kubernetes.io/projected/da10e874-0602-4ca3-86aa-500bb5b32448-kube-api-access-xdd6q\") pod \"da10e874-0602-4ca3-86aa-500bb5b32448\" (UID: \"da10e874-0602-4ca3-86aa-500bb5b32448\") " Oct 31 22:14:28.162495 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.152542 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da10e874-0602-4ca3-86aa-500bb5b32448-openshift-service-ca" pod "da10e874-0602-4ca3-86aa-500bb5b32448" (UID: "da10e874-0602-4ca3-86aa-500bb5b32448"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:14:28.162495 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.158614 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-metrics-certs" pod "da10e874-0602-4ca3-86aa-500bb5b32448" (UID: "da10e874-0602-4ca3-86aa-500bb5b32448"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:14:28.162495 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.158834 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da10e874-0602-4ca3-86aa-500bb5b32448-kube-api-access-xdd6q" pod "da10e874-0602-4ca3-86aa-500bb5b32448" (UID: "da10e874-0602-4ca3-86aa-500bb5b32448"). InnerVolumeSpecName "kube-api-access-xdd6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:14:28.162495 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.161517 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-maas-api-tls" pod "da10e874-0602-4ca3-86aa-500bb5b32448" (UID: "da10e874-0602-4ca3-86aa-500bb5b32448"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:14:28.246737 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.246692 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:14:28.246737 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.246734 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdd6q\" (UniqueName: \"kubernetes.io/projected/da10e874-0602-4ca3-86aa-500bb5b32448-kube-api-access-xdd6q\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:14:28.246979 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.246748 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da10e874-0602-4ca3-86aa-500bb5b32448-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:14:28.246979 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.246760 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da10e874-0602-4ca3-86aa-500bb5b32448-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:14:28.750162 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.750127 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" Oct 31 22:14:28.750162 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.750151 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf" event={"ID":"da10e874-0602-4ca3-86aa-500bb5b32448","Type":"ContainerDied","Data":"941678448ee289a5a9b804c9b295addc0f7da3a4525328cb674691205f5e69ed"} Oct 31 22:14:28.750781 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.750216 5188 scope.go:122] "RemoveContainer" containerID="24491d40dc0d767cfa5e32d8e8b738d439858dea3949dff2cb4008e6c3db85fc" Oct 31 22:14:28.801994 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.801958 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf"] Oct 31 22:14:28.811901 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:28.811858 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4275b9-fc995dcd4-cr5nf"] Oct 31 22:14:29.481173 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:29.481128 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da10e874-0602-4ca3-86aa-500bb5b32448" path="/var/lib/kubelet/pods/da10e874-0602-4ca3-86aa-500bb5b32448/volumes" Oct 31 22:14:32.540932 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:14:32.540893 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:14:46.734887 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:46.734834 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw"] Oct 31 22:14:51.736186 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:51.736140 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" podUID="95a84925-746c-4331-8f5a-197e34cd274d" containerName="payload-processing" containerID="cri-o://fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914" gracePeriod=25 Oct 31 22:14:59.443288 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:14:59.443244 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" podUID="95a84925-746c-4331-8f5a-197e34cd274d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:15:02.541364 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:15:02.541304 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:15:09.443468 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:09.443370 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" podUID="95a84925-746c-4331-8f5a-197e34cd274d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:15:12.040747 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.040709 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 22:15:12.120210 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.120173 5188 generic.go:356] "Generic (PLEG): container finished" podID="95a84925-746c-4331-8f5a-197e34cd274d" containerID="fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914" exitCode=0 Oct 31 22:15:12.120443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.120234 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" Oct 31 22:15:12.120443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.120263 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" event={"ID":"95a84925-746c-4331-8f5a-197e34cd274d","Type":"ContainerDied","Data":"fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914"} Oct 31 22:15:12.120443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.120378 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw" event={"ID":"95a84925-746c-4331-8f5a-197e34cd274d","Type":"ContainerDied","Data":"630d387fb928da3ab68622b39963a18882d0ffcd4fd6dbcd47aa322952ecf6c9"} Oct 31 22:15:12.120443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.120419 5188 scope.go:122] "RemoveContainer" containerID="fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914" Oct 31 22:15:12.137548 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.137523 5188 scope.go:122] "RemoveContainer" containerID="fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914" Oct 31 22:15:12.137908 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:15:12.137881 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914\": container with ID starting with fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914 not found: ID does not exist" containerID="fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914" Oct 31 22:15:12.138016 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.137915 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914"} err="failed to get container status \"fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914\": rpc error: code = NotFound desc = could not find container \"fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914\": container with ID starting with fb0aa674fa8e3c6d3f164d609873ebeda2589b244fb37da8f15a37f1db633914 not found: ID does not exist" Oct 31 22:15:12.168179 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.168151 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95a84925-746c-4331-8f5a-197e34cd274d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a84925-746c-4331-8f5a-197e34cd274d-plugins-config-volume\") pod \"95a84925-746c-4331-8f5a-197e34cd274d\" (UID: \"95a84925-746c-4331-8f5a-197e34cd274d\") " Oct 31 22:15:12.168385 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.168312 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95a84925-746c-4331-8f5a-197e34cd274d-kube-api-access-4vpgh\" (UniqueName: \"kubernetes.io/projected/95a84925-746c-4331-8f5a-197e34cd274d-kube-api-access-4vpgh\") pod \"95a84925-746c-4331-8f5a-197e34cd274d\" (UID: \"95a84925-746c-4331-8f5a-197e34cd274d\") " Oct 31 22:15:12.168619 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.168591 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95a84925-746c-4331-8f5a-197e34cd274d-plugins-config-volume" pod "95a84925-746c-4331-8f5a-197e34cd274d" (UID: "95a84925-746c-4331-8f5a-197e34cd274d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:12.168818 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.168803 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95a84925-746c-4331-8f5a-197e34cd274d-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:12.170382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.170356 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95a84925-746c-4331-8f5a-197e34cd274d-kube-api-access-4vpgh" pod "95a84925-746c-4331-8f5a-197e34cd274d" (UID: "95a84925-746c-4331-8f5a-197e34cd274d"). InnerVolumeSpecName "kube-api-access-4vpgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:12.269951 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.269856 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vpgh\" (UniqueName: \"kubernetes.io/projected/95a84925-746c-4331-8f5a-197e34cd274d-kube-api-access-4vpgh\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:12.463354 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.463281 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw"] Oct 31 22:15:12.470189 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:12.470148 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a5affe-86cc9766b5-278sw"] Oct 31 22:15:13.476476 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:13.476438 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95a84925-746c-4331-8f5a-197e34cd274d" path="/var/lib/kubelet/pods/95a84925-746c-4331-8f5a-197e34cd274d/volumes" Oct 31 22:15:14.807525 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:14.807481 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn"] Oct 31 22:15:14.808043 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:14.807792 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" podUID="83e142d5-6162-47b4-9f68-c1c16aa8a778" containerName="maas-api" containerID="cri-o://ae2ecb72d4fbd2392d93d1304b7a0447070e606b7804c28323a76845f4b79f1e" gracePeriod=30 Oct 31 22:15:15.166979 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.166289 5188 generic.go:356] "Generic (PLEG): container finished" podID="83e142d5-6162-47b4-9f68-c1c16aa8a778" containerID="ae2ecb72d4fbd2392d93d1304b7a0447070e606b7804c28323a76845f4b79f1e" exitCode=0 Oct 31 22:15:15.166979 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.166546 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" event={"ID":"83e142d5-6162-47b4-9f68-c1c16aa8a778","Type":"ContainerDied","Data":"ae2ecb72d4fbd2392d93d1304b7a0447070e606b7804c28323a76845f4b79f1e"} Oct 31 22:15:15.166979 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.166595 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" event={"ID":"83e142d5-6162-47b4-9f68-c1c16aa8a778","Type":"ContainerDied","Data":"269e520eca3c9647c5037331f65bda5580ad48fa93a9b263466e033f0a708167"} Oct 31 22:15:15.166979 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.166614 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="269e520eca3c9647c5037331f65bda5580ad48fa93a9b263466e033f0a708167" Oct 31 22:15:15.177641 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.177607 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 22:15:15.321476 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.321438 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e142d5-6162-47b4-9f68-c1c16aa8a778-kube-api-access-nc924\" (UniqueName: \"kubernetes.io/projected/83e142d5-6162-47b4-9f68-c1c16aa8a778-kube-api-access-nc924\") pod \"83e142d5-6162-47b4-9f68-c1c16aa8a778\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " Oct 31 22:15:15.321806 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.321781 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e142d5-6162-47b4-9f68-c1c16aa8a778-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e142d5-6162-47b4-9f68-c1c16aa8a778-openshift-service-ca\") pod \"83e142d5-6162-47b4-9f68-c1c16aa8a778\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " Oct 31 22:15:15.323303 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.323275 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-maas-api-tls\") pod \"83e142d5-6162-47b4-9f68-c1c16aa8a778\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " Oct 31 22:15:15.323620 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.323600 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-metrics-certs\") pod \"83e142d5-6162-47b4-9f68-c1c16aa8a778\" (UID: \"83e142d5-6162-47b4-9f68-c1c16aa8a778\") " Oct 31 22:15:15.324822 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.324790 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e142d5-6162-47b4-9f68-c1c16aa8a778-openshift-service-ca" pod "83e142d5-6162-47b4-9f68-c1c16aa8a778" (UID: "83e142d5-6162-47b4-9f68-c1c16aa8a778"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:15.327480 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.327439 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-metrics-certs" pod "83e142d5-6162-47b4-9f68-c1c16aa8a778" (UID: "83e142d5-6162-47b4-9f68-c1c16aa8a778"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:15:15.327688 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.327665 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e142d5-6162-47b4-9f68-c1c16aa8a778-kube-api-access-nc924" pod "83e142d5-6162-47b4-9f68-c1c16aa8a778" (UID: "83e142d5-6162-47b4-9f68-c1c16aa8a778"). InnerVolumeSpecName "kube-api-access-nc924". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:15.329071 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.329043 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-maas-api-tls" pod "83e142d5-6162-47b4-9f68-c1c16aa8a778" (UID: "83e142d5-6162-47b4-9f68-c1c16aa8a778"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:15:15.426269 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.426168 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc924\" (UniqueName: \"kubernetes.io/projected/83e142d5-6162-47b4-9f68-c1c16aa8a778-kube-api-access-nc924\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:15.426269 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.426213 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e142d5-6162-47b4-9f68-c1c16aa8a778-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:15.426269 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.426230 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:15.426269 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:15.426245 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83e142d5-6162-47b4-9f68-c1c16aa8a778-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:16.174626 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:16.174584 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn" Oct 31 22:15:16.221014 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:16.220977 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn"] Oct 31 22:15:16.233198 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:16.233159 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5affe-784bb76d-jckjn"] Oct 31 22:15:17.035505 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:17.035441 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4"] Oct 31 22:15:17.475173 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:17.475140 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83e142d5-6162-47b4-9f68-c1c16aa8a778" path="/var/lib/kubelet/pods/83e142d5-6162-47b4-9f68-c1c16aa8a778/volumes" Oct 31 22:15:32.061925 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.061853 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" podUID="db87ebb5-bb07-4598-b6e0-c3d587a05cbf" containerName="main" containerID="cri-o://f0f2d5c0c64cc2f8982cc92da47a892b409a4a0daa6904b39c3da556fb51dc96" gracePeriod=45 Oct 31 22:15:32.316134 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.316031 5188 generic.go:356] "Generic (PLEG): container finished" podID="db87ebb5-bb07-4598-b6e0-c3d587a05cbf" containerID="f0f2d5c0c64cc2f8982cc92da47a892b409a4a0daa6904b39c3da556fb51dc96" exitCode=0 Oct 31 22:15:32.316134 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.316080 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" event={"ID":"db87ebb5-bb07-4598-b6e0-c3d587a05cbf","Type":"ContainerDied","Data":"f0f2d5c0c64cc2f8982cc92da47a892b409a4a0daa6904b39c3da556fb51dc96"} Oct 31 22:15:32.366600 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.366572 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:15:32.473678 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.473639 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-kube-api-access-wwcdn\" (UniqueName: \"kubernetes.io/projected/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-kube-api-access-wwcdn\") pod \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " Oct 31 22:15:32.473916 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.473701 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-model-cache\") pod \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " Oct 31 22:15:32.473916 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.473812 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-dshm\") pod \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " Oct 31 22:15:32.473916 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.473910 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-home\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-home\") pod \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " Oct 31 22:15:32.474116 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.473945 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tls-certs\") pod \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " Oct 31 22:15:32.474116 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.473984 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tmp-dir\") pod \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\" (UID: \"db87ebb5-bb07-4598-b6e0-c3d587a05cbf\") " Oct 31 22:15:32.475478 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.475064 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tmp-dir" pod "db87ebb5-bb07-4598-b6e0-c3d587a05cbf" (UID: "db87ebb5-bb07-4598-b6e0-c3d587a05cbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:32.475478 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.475446 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:32.475860 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.475832 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-model-cache" pod "db87ebb5-bb07-4598-b6e0-c3d587a05cbf" (UID: "db87ebb5-bb07-4598-b6e0-c3d587a05cbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:32.476107 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.476086 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-home" pod "db87ebb5-bb07-4598-b6e0-c3d587a05cbf" (UID: "db87ebb5-bb07-4598-b6e0-c3d587a05cbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:32.477988 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.477955 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tls-certs" pod "db87ebb5-bb07-4598-b6e0-c3d587a05cbf" (UID: "db87ebb5-bb07-4598-b6e0-c3d587a05cbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:15:32.480166 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.480126 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-dshm" pod "db87ebb5-bb07-4598-b6e0-c3d587a05cbf" (UID: "db87ebb5-bb07-4598-b6e0-c3d587a05cbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:32.480775 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.480749 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-kube-api-access-wwcdn" pod "db87ebb5-bb07-4598-b6e0-c3d587a05cbf" (UID: "db87ebb5-bb07-4598-b6e0-c3d587a05cbf"). InnerVolumeSpecName "kube-api-access-wwcdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:32.544566 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:15:32.544517 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:15:32.577256 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.577153 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:32.577256 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.577200 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:32.577256 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.577213 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:32.577256 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.577227 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:32.577256 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:32.577240 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwcdn\" (UniqueName: \"kubernetes.io/projected/db87ebb5-bb07-4598-b6e0-c3d587a05cbf-kube-api-access-wwcdn\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:33.326655 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:33.326619 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" Oct 31 22:15:33.327164 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:33.326650 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4" event={"ID":"db87ebb5-bb07-4598-b6e0-c3d587a05cbf","Type":"ContainerDied","Data":"8d400d6d25fa1f4b5d5e99c5bceb2060601c61eb866a6fb9b5736eacaaa053dc"} Oct 31 22:15:33.327164 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:33.326719 5188 scope.go:122] "RemoveContainer" containerID="f0f2d5c0c64cc2f8982cc92da47a892b409a4a0daa6904b39c3da556fb51dc96" Oct 31 22:15:33.378670 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:33.378625 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4"] Oct 31 22:15:33.388498 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:33.388462 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c2d1b8/test-model-c2d1b8-kserve-db65b8447-jlpr4"] Oct 31 22:15:33.475819 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:33.475778 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db87ebb5-bb07-4598-b6e0-c3d587a05cbf" path="/var/lib/kubelet/pods/db87ebb5-bb07-4598-b6e0-c3d587a05cbf/volumes" Oct 31 22:15:42.646056 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:42.646011 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7"] Oct 31 22:15:42.646582 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:42.646252 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" podUID="853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" containerName="maas-api" containerID="cri-o://7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47" gracePeriod=30 Oct 31 22:15:43.232061 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.232032 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:15:43.347748 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.347715 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-openshift-service-ca\") pod \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " Oct 31 22:15:43.347950 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.347823 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-maas-api-tls\") pod \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " Oct 31 22:15:43.347950 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.347881 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-metrics-certs\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-metrics-certs\") pod \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " Oct 31 22:15:43.347950 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.347916 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-kube-api-access-55j8g\" (UniqueName: \"kubernetes.io/projected/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-kube-api-access-55j8g\") pod \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\" (UID: \"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456\") " Oct 31 22:15:43.348174 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.348150 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-openshift-service-ca" pod "853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" (UID: "853d5e9a-0ff1-4dd9-be7c-efda8dfaa456"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:43.348543 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.348520 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:43.350138 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.350107 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-maas-api-tls" pod "853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" (UID: "853d5e9a-0ff1-4dd9-be7c-efda8dfaa456"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:15:43.350253 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.350173 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-kube-api-access-55j8g" pod "853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" (UID: "853d5e9a-0ff1-4dd9-be7c-efda8dfaa456"). InnerVolumeSpecName "kube-api-access-55j8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:43.350253 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.350180 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-metrics-certs" pod "853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" (UID: "853d5e9a-0ff1-4dd9-be7c-efda8dfaa456"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:15:43.414255 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.414215 5188 generic.go:356] "Generic (PLEG): container finished" podID="853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" containerID="7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47" exitCode=0 Oct 31 22:15:43.414516 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.414276 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" Oct 31 22:15:43.414516 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.414297 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" event={"ID":"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456","Type":"ContainerDied","Data":"7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47"} Oct 31 22:15:43.414516 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.414378 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7" event={"ID":"853d5e9a-0ff1-4dd9-be7c-efda8dfaa456","Type":"ContainerDied","Data":"4b8f3aab8b5a306268c2adfce4d350241a103ff48960f7aa81cc14c794479969"} Oct 31 22:15:43.414516 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.414415 5188 scope.go:122] "RemoveContainer" containerID="7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47" Oct 31 22:15:43.427914 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.427882 5188 scope.go:122] "RemoveContainer" containerID="7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47" Oct 31 22:15:43.428247 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:15:43.428224 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47\": container with ID starting with 7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47 not found: ID does not exist" containerID="7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47" Oct 31 22:15:43.428308 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.428258 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47"} err="failed to get container status \"7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47\": rpc error: code = NotFound desc = could not find container \"7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47\": container with ID starting with 7fbc9575e97c6bb58e63c2a246308ffba004d3c1b790be4ac91f1f3373135f47 not found: ID does not exist" Oct 31 22:15:43.449387 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.449262 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:43.449387 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.449296 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:43.449387 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.449309 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55j8g\" (UniqueName: \"kubernetes.io/projected/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456-kube-api-access-55j8g\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:15:43.464652 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.464586 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7"] Oct 31 22:15:43.476190 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:43.476158 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-710927-7d44765d85-7qrf7"] Oct 31 22:15:45.474538 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:45.474502 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" path="/var/lib/kubelet/pods/853d5e9a-0ff1-4dd9-be7c-efda8dfaa456/volumes" Oct 31 22:15:54.214809 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:54.214770 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-k4qxr_e49a2d50-dc82-4983-9425-053dfe903ddd/cluster-samples-operator/0.log" Oct 31 22:15:54.217895 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:54.217861 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:15:54.222274 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:54.222246 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-k4qxr_e49a2d50-dc82-4983-9425-053dfe903ddd/cluster-samples-operator/0.log" Oct 31 22:15:54.225310 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:54.225283 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:15:57.201518 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:15:57.201475 5188 scope.go:122] "RemoveContainer" containerID="ae2ecb72d4fbd2392d93d1304b7a0447070e606b7804c28323a76845f4b79f1e" Oct 31 22:16:02.550482 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:16:02.550438 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:16:04.750179 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:04.750140 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78"] Oct 31 22:16:04.750760 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:04.750406 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" podUID="abb5e084-18e0-47d2-85f7-a5102d5a8055" containerName="maas-api" containerID="cri-o://f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020" gracePeriod=30 Oct 31 22:16:05.130268 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.130233 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 22:16:05.196113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.195578 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb5e084-18e0-47d2-85f7-a5102d5a8055-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb5e084-18e0-47d2-85f7-a5102d5a8055-openshift-service-ca\") pod \"abb5e084-18e0-47d2-85f7-a5102d5a8055\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " Oct 31 22:16:05.196113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.195661 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb5e084-18e0-47d2-85f7-a5102d5a8055-kube-api-access-vf829\" (UniqueName: \"kubernetes.io/projected/abb5e084-18e0-47d2-85f7-a5102d5a8055-kube-api-access-vf829\") pod \"abb5e084-18e0-47d2-85f7-a5102d5a8055\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " Oct 31 22:16:05.196113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.195729 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-metrics-certs\") pod \"abb5e084-18e0-47d2-85f7-a5102d5a8055\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " Oct 31 22:16:05.196113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.195840 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-maas-api-tls\") pod \"abb5e084-18e0-47d2-85f7-a5102d5a8055\" (UID: \"abb5e084-18e0-47d2-85f7-a5102d5a8055\") " Oct 31 22:16:05.201766 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.201722 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-maas-api-tls" pod "abb5e084-18e0-47d2-85f7-a5102d5a8055" (UID: "abb5e084-18e0-47d2-85f7-a5102d5a8055"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:05.202740 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.202705 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb5e084-18e0-47d2-85f7-a5102d5a8055-openshift-service-ca" pod "abb5e084-18e0-47d2-85f7-a5102d5a8055" (UID: "abb5e084-18e0-47d2-85f7-a5102d5a8055"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:05.204054 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.204016 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb5e084-18e0-47d2-85f7-a5102d5a8055-kube-api-access-vf829" pod "abb5e084-18e0-47d2-85f7-a5102d5a8055" (UID: "abb5e084-18e0-47d2-85f7-a5102d5a8055"). InnerVolumeSpecName "kube-api-access-vf829". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:05.204295 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.204274 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-metrics-certs" pod "abb5e084-18e0-47d2-85f7-a5102d5a8055" (UID: "abb5e084-18e0-47d2-85f7-a5102d5a8055"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:05.298135 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.298090 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abb5e084-18e0-47d2-85f7-a5102d5a8055-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:05.298135 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.298136 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf829\" (UniqueName: \"kubernetes.io/projected/abb5e084-18e0-47d2-85f7-a5102d5a8055-kube-api-access-vf829\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:05.298390 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.298154 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:05.298390 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.298168 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abb5e084-18e0-47d2-85f7-a5102d5a8055-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:05.662057 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.662005 5188 generic.go:356] "Generic (PLEG): container finished" podID="abb5e084-18e0-47d2-85f7-a5102d5a8055" containerID="f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020" exitCode=0 Oct 31 22:16:05.662264 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.662229 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" event={"ID":"abb5e084-18e0-47d2-85f7-a5102d5a8055","Type":"ContainerDied","Data":"f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020"} Oct 31 22:16:05.662392 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.662367 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" event={"ID":"abb5e084-18e0-47d2-85f7-a5102d5a8055","Type":"ContainerDied","Data":"24fda4731b78eda20f90e2d9ba81b19a2a88eaf5ff49786c57e1ca45fa8e3fd8"} Oct 31 22:16:05.662476 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.662389 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78" Oct 31 22:16:05.662476 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.662421 5188 scope.go:122] "RemoveContainer" containerID="f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020" Oct 31 22:16:05.677168 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.677140 5188 scope.go:122] "RemoveContainer" containerID="f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020" Oct 31 22:16:05.677777 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:16:05.677743 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020\": container with ID starting with f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020 not found: ID does not exist" containerID="f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020" Oct 31 22:16:05.677888 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.677790 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020"} err="failed to get container status \"f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020\": rpc error: code = NotFound desc = could not find container \"f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020\": container with ID starting with f0085ec5165082b8154a09af57941d08ec9ddbc6f13a8f1b94990e7795c43020 not found: ID does not exist" Oct 31 22:16:05.710847 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.710804 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78"] Oct 31 22:16:05.719739 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:05.719705 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b5dab-d584ff4f7-bgk78"] Oct 31 22:16:07.473881 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:07.473845 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb5e084-18e0-47d2-85f7-a5102d5a8055" path="/var/lib/kubelet/pods/abb5e084-18e0-47d2-85f7-a5102d5a8055/volumes" Oct 31 22:16:23.949474 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:23.949437 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm"] Oct 31 22:16:28.461629 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.461586 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm"] Oct 31 22:16:28.462094 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.461867 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" podUID="68f1815c-225d-4141-9279-eb765de32aa3" containerName="maas-api" containerID="cri-o://040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e" gracePeriod=30 Oct 31 22:16:28.775775 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.775738 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx"] Oct 31 22:16:28.776223 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.776187 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" containerID="cri-o://431a263cd0f1114c730a30f63e5a0a6e3e1ae2f1b2a4e48fa10f81df3284a884" gracePeriod=30 Oct 31 22:16:28.805509 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.799958 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb"] Oct 31 22:16:28.805509 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.800223 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" containerID="cri-o://eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02" gracePeriod=30 Oct 31 22:16:28.827853 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.827816 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:16:28.874010 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.873965 5188 generic.go:356] "Generic (PLEG): container finished" podID="68f1815c-225d-4141-9279-eb765de32aa3" containerID="040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e" exitCode=0 Oct 31 22:16:28.874221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.874159 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" event={"ID":"68f1815c-225d-4141-9279-eb765de32aa3","Type":"ContainerDied","Data":"040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e"} Oct 31 22:16:28.874221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.874213 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" event={"ID":"68f1815c-225d-4141-9279-eb765de32aa3","Type":"ContainerDied","Data":"a66bc3cf8a2b2d9bbb18e2a5b12f6f9c5583aea9ddc00d44ffc0b50b8b5eb9d8"} Oct 31 22:16:28.874393 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.874244 5188 scope.go:122] "RemoveContainer" containerID="040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e" Oct 31 22:16:28.874619 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.874470 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm" Oct 31 22:16:28.898540 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.898505 5188 scope.go:122] "RemoveContainer" containerID="040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e" Oct 31 22:16:28.899001 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:16:28.898972 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e\": container with ID starting with 040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e not found: ID does not exist" containerID="040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e" Oct 31 22:16:28.899127 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.899015 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e"} err="failed to get container status \"040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e\": rpc error: code = NotFound desc = could not find container \"040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e\": container with ID starting with 040f2c0f74951ed36d46005dbee3d3746bb02e9864068ed19e8f602d0271009e not found: ID does not exist" Oct 31 22:16:28.941703 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.941662 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-metrics-certs\") pod \"68f1815c-225d-4141-9279-eb765de32aa3\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " Oct 31 22:16:28.941913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.941769 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f1815c-225d-4141-9279-eb765de32aa3-kube-api-access-7hdhz\" (UniqueName: \"kubernetes.io/projected/68f1815c-225d-4141-9279-eb765de32aa3-kube-api-access-7hdhz\") pod \"68f1815c-225d-4141-9279-eb765de32aa3\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " Oct 31 22:16:28.941913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.941878 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-maas-api-tls\") pod \"68f1815c-225d-4141-9279-eb765de32aa3\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " Oct 31 22:16:28.942060 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.941981 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f1815c-225d-4141-9279-eb765de32aa3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f1815c-225d-4141-9279-eb765de32aa3-openshift-service-ca\") pod \"68f1815c-225d-4141-9279-eb765de32aa3\" (UID: \"68f1815c-225d-4141-9279-eb765de32aa3\") " Oct 31 22:16:28.942639 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.942589 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f1815c-225d-4141-9279-eb765de32aa3-openshift-service-ca" pod "68f1815c-225d-4141-9279-eb765de32aa3" (UID: "68f1815c-225d-4141-9279-eb765de32aa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:28.944577 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.944544 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f1815c-225d-4141-9279-eb765de32aa3-kube-api-access-7hdhz" pod "68f1815c-225d-4141-9279-eb765de32aa3" (UID: "68f1815c-225d-4141-9279-eb765de32aa3"). InnerVolumeSpecName "kube-api-access-7hdhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:28.944711 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.944619 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-metrics-certs" pod "68f1815c-225d-4141-9279-eb765de32aa3" (UID: "68f1815c-225d-4141-9279-eb765de32aa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:28.945268 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.945248 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-maas-api-tls" pod "68f1815c-225d-4141-9279-eb765de32aa3" (UID: "68f1815c-225d-4141-9279-eb765de32aa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:28.950407 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:28.950372 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" podUID="931df1cf-f6d0-4c1e-b87f-e596aba68266" containerName="payload-pre-processing" containerID="cri-o://169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697" gracePeriod=25 Oct 31 22:16:29.050401 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:29.044391 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68f1815c-225d-4141-9279-eb765de32aa3-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:29.050401 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:29.044451 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:29.050401 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:29.044470 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hdhz\" (UniqueName: \"kubernetes.io/projected/68f1815c-225d-4141-9279-eb765de32aa3-kube-api-access-7hdhz\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:29.050401 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:29.044486 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68f1815c-225d-4141-9279-eb765de32aa3-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:29.223682 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:29.223637 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm"] Oct 31 22:16:29.227022 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:29.226987 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c2d1b8-56564cfd54-k89gm"] Oct 31 22:16:29.474159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:29.474120 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f1815c-225d-4141-9279-eb765de32aa3" path="/var/lib/kubelet/pods/68f1815c-225d-4141-9279-eb765de32aa3/volumes" Oct 31 22:16:30.972680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:30.972637 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" podUID="931df1cf-f6d0-4c1e-b87f-e596aba68266" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:16:32.540011 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:16:32.539969 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:16:33.767445 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:33.767406 5188 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:16:33.767890 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:33.767478 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:16:33.914657 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:33.914579 5188 generic.go:356] "Generic (PLEG): container finished" podID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerID="431a263cd0f1114c730a30f63e5a0a6e3e1ae2f1b2a4e48fa10f81df3284a884" exitCode=0 Oct 31 22:16:33.914657 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:33.914642 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" event={"ID":"bd42e3ee-f036-4bc4-87ab-45103198c952","Type":"ContainerDied","Data":"431a263cd0f1114c730a30f63e5a0a6e3e1ae2f1b2a4e48fa10f81df3284a884"} Oct 31 22:16:34.120350 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.119819 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 22:16:34.159075 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.159043 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:16:34.218072 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218026 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-token\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-token\") pod \"36a019eb-d5f0-41f8-9822-eeb0c265e906\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " Oct 31 22:16:34.218287 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218109 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-certs\") pod \"bd42e3ee-f036-4bc4-87ab-45103198c952\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " Oct 31 22:16:34.218287 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218150 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-envoy\") pod \"bd42e3ee-f036-4bc4-87ab-45103198c952\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " Oct 31 22:16:34.218622 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218593 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-data\") pod \"36a019eb-d5f0-41f8-9822-eeb0c265e906\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " Oct 31 22:16:34.218695 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218601 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-certs" pod "bd42e3ee-f036-4bc4-87ab-45103198c952" (UID: "bd42e3ee-f036-4bc4-87ab-45103198c952"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.218695 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218668 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-podinfo\") pod \"36a019eb-d5f0-41f8-9822-eeb0c265e906\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " Oct 31 22:16:34.218811 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218701 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-envoy\") pod \"36a019eb-d5f0-41f8-9822-eeb0c265e906\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " Oct 31 22:16:34.218811 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218760 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-credential-socket\") pod \"bd42e3ee-f036-4bc4-87ab-45103198c952\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " Oct 31 22:16:34.218811 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218788 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-credential-socket\") pod \"36a019eb-d5f0-41f8-9822-eeb0c265e906\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " Oct 31 22:16:34.218985 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218828 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-kube-api-access-c5rdm\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-kube-api-access-c5rdm\") pod \"36a019eb-d5f0-41f8-9822-eeb0c265e906\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " Oct 31 22:16:34.218985 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.218860 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-socket\") pod \"36a019eb-d5f0-41f8-9822-eeb0c265e906\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " Oct 31 22:16:34.219317 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219291 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-certs\") pod \"36a019eb-d5f0-41f8-9822-eeb0c265e906\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " Oct 31 22:16:34.219443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219356 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a019eb-d5f0-41f8-9822-eeb0c265e906-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a019eb-d5f0-41f8-9822-eeb0c265e906-istiod-ca-cert\") pod \"36a019eb-d5f0-41f8-9822-eeb0c265e906\" (UID: \"36a019eb-d5f0-41f8-9822-eeb0c265e906\") " Oct 31 22:16:34.219443 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219404 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-kube-api-access-f2mkf\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-kube-api-access-f2mkf\") pod \"bd42e3ee-f036-4bc4-87ab-45103198c952\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " Oct 31 22:16:34.219566 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219452 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-podinfo\") pod \"bd42e3ee-f036-4bc4-87ab-45103198c952\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " Oct 31 22:16:34.219629 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219566 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-socket\") pod \"bd42e3ee-f036-4bc4-87ab-45103198c952\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " Oct 31 22:16:34.219686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219629 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-data\") pod \"bd42e3ee-f036-4bc4-87ab-45103198c952\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " Oct 31 22:16:34.219686 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219672 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd42e3ee-f036-4bc4-87ab-45103198c952-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd42e3ee-f036-4bc4-87ab-45103198c952-istiod-ca-cert\") pod \"bd42e3ee-f036-4bc4-87ab-45103198c952\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " Oct 31 22:16:34.219790 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219708 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-token\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-token\") pod \"bd42e3ee-f036-4bc4-87ab-45103198c952\" (UID: \"bd42e3ee-f036-4bc4-87ab-45103198c952\") " Oct 31 22:16:34.219790 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219753 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-socket" pod "36a019eb-d5f0-41f8-9822-eeb0c265e906" (UID: "36a019eb-d5f0-41f8-9822-eeb0c265e906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.220479 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.219990 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-credential-socket" pod "bd42e3ee-f036-4bc4-87ab-45103198c952" (UID: "bd42e3ee-f036-4bc4-87ab-45103198c952"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.220479 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.220127 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-data" pod "36a019eb-d5f0-41f8-9822-eeb0c265e906" (UID: "36a019eb-d5f0-41f8-9822-eeb0c265e906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.220479 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.220183 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-credential-socket" pod "36a019eb-d5f0-41f8-9822-eeb0c265e906" (UID: "36a019eb-d5f0-41f8-9822-eeb0c265e906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.223489 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.223455 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a019eb-d5f0-41f8-9822-eeb0c265e906-istiod-ca-cert" pod "36a019eb-d5f0-41f8-9822-eeb0c265e906" (UID: "36a019eb-d5f0-41f8-9822-eeb0c265e906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:34.223750 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.223724 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-certs" pod "36a019eb-d5f0-41f8-9822-eeb0c265e906" (UID: "36a019eb-d5f0-41f8-9822-eeb0c265e906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.223994 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.223972 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-data" pod "bd42e3ee-f036-4bc4-87ab-45103198c952" (UID: "bd42e3ee-f036-4bc4-87ab-45103198c952"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.224158 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.224139 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-socket" pod "bd42e3ee-f036-4bc4-87ab-45103198c952" (UID: "bd42e3ee-f036-4bc4-87ab-45103198c952"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.224460 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.224436 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd42e3ee-f036-4bc4-87ab-45103198c952-istiod-ca-cert" pod "bd42e3ee-f036-4bc4-87ab-45103198c952" (UID: "bd42e3ee-f036-4bc4-87ab-45103198c952"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:34.227791 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.227758 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-kube-api-access-f2mkf" pod "bd42e3ee-f036-4bc4-87ab-45103198c952" (UID: "bd42e3ee-f036-4bc4-87ab-45103198c952"). InnerVolumeSpecName "kube-api-access-f2mkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:34.227963 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.227933 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.227963 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.227958 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.228136 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.227973 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.228136 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.227986 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.228136 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.228002 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.228136 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.228016 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36a019eb-d5f0-41f8-9822-eeb0c265e906-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.228136 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.228030 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.228136 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.228042 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.228136 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.228055 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.228573 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.228547 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-token" pod "bd42e3ee-f036-4bc4-87ab-45103198c952" (UID: "bd42e3ee-f036-4bc4-87ab-45103198c952"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:34.230551 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.230525 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-podinfo" pod "bd42e3ee-f036-4bc4-87ab-45103198c952" (UID: "bd42e3ee-f036-4bc4-87ab-45103198c952"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:16:34.230670 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.230615 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-kube-api-access-c5rdm" pod "36a019eb-d5f0-41f8-9822-eeb0c265e906" (UID: "36a019eb-d5f0-41f8-9822-eeb0c265e906"). InnerVolumeSpecName "kube-api-access-c5rdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:34.231599 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.231556 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-podinfo" pod "36a019eb-d5f0-41f8-9822-eeb0c265e906" (UID: "36a019eb-d5f0-41f8-9822-eeb0c265e906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:16:34.232457 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.232431 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-token" pod "36a019eb-d5f0-41f8-9822-eeb0c265e906" (UID: "36a019eb-d5f0-41f8-9822-eeb0c265e906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:34.232566 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.232466 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-envoy" pod "36a019eb-d5f0-41f8-9822-eeb0c265e906" (UID: "36a019eb-d5f0-41f8-9822-eeb0c265e906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.232913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.232891 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-envoy" pod "bd42e3ee-f036-4bc4-87ab-45103198c952" (UID: "bd42e3ee-f036-4bc4-87ab-45103198c952"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.329680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.329587 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.329680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.329625 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.329680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.329635 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5rdm\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-kube-api-access-c5rdm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.329680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.329644 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2mkf\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-kube-api-access-f2mkf\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.329680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.329654 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.329680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.329663 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd42e3ee-f036-4bc4-87ab-45103198c952-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.329680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.329673 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.329680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.329684 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36a019eb-d5f0-41f8-9822-eeb0c265e906-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.330231 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.329692 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd42e3ee-f036-4bc4-87ab-45103198c952-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.924596 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.924556 5188 generic.go:356] "Generic (PLEG): container finished" podID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerID="eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02" exitCode=0 Oct 31 22:16:34.925068 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.924604 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" Oct 31 22:16:34.925068 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.924637 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" event={"ID":"36a019eb-d5f0-41f8-9822-eeb0c265e906","Type":"ContainerDied","Data":"eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02"} Oct 31 22:16:34.925068 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.924696 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb" event={"ID":"36a019eb-d5f0-41f8-9822-eeb0c265e906","Type":"ContainerDied","Data":"ff73a29a44decb1ae28c756009c61746b2c7cac4cc850890a2c44ede85503f36"} Oct 31 22:16:34.925068 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.924728 5188 scope.go:122] "RemoveContainer" containerID="eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02" Oct 31 22:16:34.926736 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.926706 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" Oct 31 22:16:34.926895 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.926730 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx" event={"ID":"bd42e3ee-f036-4bc4-87ab-45103198c952","Type":"ContainerDied","Data":"c48ddd60905c93b623c69333f59cfb94e1cb773702d7ff387ebefe205fd49788"} Oct 31 22:16:34.938905 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.938750 5188 scope.go:122] "RemoveContainer" containerID="eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02" Oct 31 22:16:34.939129 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:16:34.939107 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02\": container with ID starting with eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02 not found: ID does not exist" containerID="eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02" Oct 31 22:16:34.939172 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.939141 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02"} err="failed to get container status \"eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02\": rpc error: code = NotFound desc = could not find container \"eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02\": container with ID starting with eedb52e2739ced5fc3aad939b709a4bf7a113ae5e1b3203bb8093f31e91ced02 not found: ID does not exist" Oct 31 22:16:34.939172 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.939160 5188 scope.go:122] "RemoveContainer" containerID="431a263cd0f1114c730a30f63e5a0a6e3e1ae2f1b2a4e48fa10f81df3284a884" Oct 31 22:16:34.974869 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.974820 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb"] Oct 31 22:16:34.984221 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:34.984177 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gjgbb"] Oct 31 22:16:35.022556 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:35.022513 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx"] Oct 31 22:16:35.027110 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:35.027076 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-td8fx"] Oct 31 22:16:35.474262 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:35.474225 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" path="/var/lib/kubelet/pods/36a019eb-d5f0-41f8-9822-eeb0c265e906/volumes" Oct 31 22:16:35.474778 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:35.474760 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" path="/var/lib/kubelet/pods/bd42e3ee-f036-4bc4-87ab-45103198c952/volumes" Oct 31 22:16:36.097519 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:36.097483 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh"] Oct 31 22:16:36.098099 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:36.097697 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" containerName="istio-proxy" containerID="cri-o://f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583" gracePeriod=30 Oct 31 22:16:39.801242 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:39.801205 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn"] Oct 31 22:16:39.801790 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:39.801472 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" containerID="cri-o://ed8c56ab5935075e85067f22a19f21a47911e7e3a18de4172d17f2ed79241c48" gracePeriod=30 Oct 31 22:16:40.973722 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:40.973679 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" podUID="931df1cf-f6d0-4c1e-b87f-e596aba68266" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:16:41.378777 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.378745 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 22:16:41.524344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524290 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4c89147-f74d-4df3-b722-48fbf418de11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c89147-f74d-4df3-b722-48fbf418de11-istio-podinfo\") pod \"d4c89147-f74d-4df3-b722-48fbf418de11\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " Oct 31 22:16:41.524580 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524363 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-socket\") pod \"d4c89147-f74d-4df3-b722-48fbf418de11\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " Oct 31 22:16:41.524580 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524425 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-envoy\") pod \"d4c89147-f74d-4df3-b722-48fbf418de11\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " Oct 31 22:16:41.524580 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524475 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-istio-token\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-istio-token\") pod \"d4c89147-f74d-4df3-b722-48fbf418de11\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " Oct 31 22:16:41.524580 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524572 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-certs\") pod \"d4c89147-f74d-4df3-b722-48fbf418de11\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " Oct 31 22:16:41.524830 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524652 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c89147-f74d-4df3-b722-48fbf418de11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c89147-f74d-4df3-b722-48fbf418de11-istiod-ca-cert\") pod \"d4c89147-f74d-4df3-b722-48fbf418de11\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " Oct 31 22:16:41.524830 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524778 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-socket" pod "d4c89147-f74d-4df3-b722-48fbf418de11" (UID: "d4c89147-f74d-4df3-b722-48fbf418de11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:41.524948 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524886 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-data\") pod \"d4c89147-f74d-4df3-b722-48fbf418de11\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " Oct 31 22:16:41.524948 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524929 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-kube-api-access-4brql\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-kube-api-access-4brql\") pod \"d4c89147-f74d-4df3-b722-48fbf418de11\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " Oct 31 22:16:41.525066 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.524999 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-credential-socket\") pod \"d4c89147-f74d-4df3-b722-48fbf418de11\" (UID: \"d4c89147-f74d-4df3-b722-48fbf418de11\") " Oct 31 22:16:41.525066 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.525043 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-certs" pod "d4c89147-f74d-4df3-b722-48fbf418de11" (UID: "d4c89147-f74d-4df3-b722-48fbf418de11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:41.525270 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.525239 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c89147-f74d-4df3-b722-48fbf418de11-istiod-ca-cert" pod "d4c89147-f74d-4df3-b722-48fbf418de11" (UID: "d4c89147-f74d-4df3-b722-48fbf418de11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:41.525451 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.525409 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-data" pod "d4c89147-f74d-4df3-b722-48fbf418de11" (UID: "d4c89147-f74d-4df3-b722-48fbf418de11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:41.525522 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.525471 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-credential-socket" pod "d4c89147-f74d-4df3-b722-48fbf418de11" (UID: "d4c89147-f74d-4df3-b722-48fbf418de11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:41.525754 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.525719 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.525854 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.525757 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.525927 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.525863 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.525927 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.525902 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.525927 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.525917 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4c89147-f74d-4df3-b722-48fbf418de11-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.528016 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.527921 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-kube-api-access-4brql" pod "d4c89147-f74d-4df3-b722-48fbf418de11" (UID: "d4c89147-f74d-4df3-b722-48fbf418de11"). InnerVolumeSpecName "kube-api-access-4brql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:41.528148 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.528036 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4c89147-f74d-4df3-b722-48fbf418de11-istio-podinfo" pod "d4c89147-f74d-4df3-b722-48fbf418de11" (UID: "d4c89147-f74d-4df3-b722-48fbf418de11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:16:41.528298 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.528274 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-istio-token" pod "d4c89147-f74d-4df3-b722-48fbf418de11" (UID: "d4c89147-f74d-4df3-b722-48fbf418de11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:41.528998 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.528971 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-envoy" pod "d4c89147-f74d-4df3-b722-48fbf418de11" (UID: "d4c89147-f74d-4df3-b722-48fbf418de11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:41.627441 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.627408 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4c89147-f74d-4df3-b722-48fbf418de11-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.627441 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.627434 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.627441 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.627443 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4brql\" (UniqueName: \"kubernetes.io/projected/d4c89147-f74d-4df3-b722-48fbf418de11-kube-api-access-4brql\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.627718 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.627455 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4c89147-f74d-4df3-b722-48fbf418de11-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.983528 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.983490 5188 generic.go:356] "Generic (PLEG): container finished" podID="d4c89147-f74d-4df3-b722-48fbf418de11" containerID="f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583" exitCode=0 Oct 31 22:16:41.984067 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.983568 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" Oct 31 22:16:41.984067 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.983588 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" event={"ID":"d4c89147-f74d-4df3-b722-48fbf418de11","Type":"ContainerDied","Data":"f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583"} Oct 31 22:16:41.984067 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.983674 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh" event={"ID":"d4c89147-f74d-4df3-b722-48fbf418de11","Type":"ContainerDied","Data":"09ec76dacd6c897e5ce25b866c9838f104b4251cd53ff5e9667bc4f756dff929"} Oct 31 22:16:41.984067 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.983688 5188 scope.go:122] "RemoveContainer" containerID="f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583" Oct 31 22:16:41.997892 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.997864 5188 scope.go:122] "RemoveContainer" containerID="f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583" Oct 31 22:16:41.998588 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:16:41.998557 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583\": container with ID starting with f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583 not found: ID does not exist" containerID="f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583" Oct 31 22:16:41.998694 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:41.998602 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583"} err="failed to get container status \"f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583\": rpc error: code = NotFound desc = could not find container \"f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583\": container with ID starting with f8207ca62acdabf26cdd30ad88e6fa31fc324cdeb99bd41e69f00d41c4a1f583 not found: ID does not exist" Oct 31 22:16:42.031117 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:42.031079 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh"] Oct 31 22:16:42.037052 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:42.037022 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-dhhgh"] Oct 31 22:16:43.474949 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:43.474913 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" path="/var/lib/kubelet/pods/d4c89147-f74d-4df3-b722-48fbf418de11/volumes" Oct 31 22:16:43.824559 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:43.824456 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9"] Oct 31 22:16:43.824767 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:43.824736 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" containerID="cri-o://0a1605db271b840aa02af7bbaee57d467b7f30469dbebca2d3591123a41d5cf6" gracePeriod=30 Oct 31 22:16:44.204169 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:44.204124 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2"] Oct 31 22:16:45.016385 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.012611 5188 generic.go:356] "Generic (PLEG): container finished" podID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerID="ed8c56ab5935075e85067f22a19f21a47911e7e3a18de4172d17f2ed79241c48" exitCode=0 Oct 31 22:16:45.016385 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.012814 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" event={"ID":"ff44385a-0c7b-48c2-b43d-a83e007ad3ae","Type":"ContainerDied","Data":"ed8c56ab5935075e85067f22a19f21a47911e7e3a18de4172d17f2ed79241c48"} Oct 31 22:16:45.122967 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.122942 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 22:16:45.170039 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170000 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-certs\") pod \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " Oct 31 22:16:45.170253 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170101 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istiod-ca-cert\") pod \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " Oct 31 22:16:45.170253 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170123 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-token\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-token\") pod \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " Oct 31 22:16:45.170253 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170157 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-data\") pod \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " Oct 31 22:16:45.170253 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170196 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-kube-api-access-82q79\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-kube-api-access-82q79\") pod \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " Oct 31 22:16:45.170253 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170237 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-envoy\") pod \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " Oct 31 22:16:45.170588 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170278 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-socket\") pod \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " Oct 31 22:16:45.170588 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170308 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-podinfo\") pod \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " Oct 31 22:16:45.170588 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170343 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-credential-socket\") pod \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\" (UID: \"ff44385a-0c7b-48c2-b43d-a83e007ad3ae\") " Oct 31 22:16:45.170588 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170312 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-certs" pod "ff44385a-0c7b-48c2-b43d-a83e007ad3ae" (UID: "ff44385a-0c7b-48c2-b43d-a83e007ad3ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:45.170588 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170518 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-data" pod "ff44385a-0c7b-48c2-b43d-a83e007ad3ae" (UID: "ff44385a-0c7b-48c2-b43d-a83e007ad3ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:45.170818 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170586 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-socket" pod "ff44385a-0c7b-48c2-b43d-a83e007ad3ae" (UID: "ff44385a-0c7b-48c2-b43d-a83e007ad3ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:45.170818 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170610 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istiod-ca-cert" pod "ff44385a-0c7b-48c2-b43d-a83e007ad3ae" (UID: "ff44385a-0c7b-48c2-b43d-a83e007ad3ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:45.170941 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170844 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:45.170941 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170866 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:45.170941 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170859 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-credential-socket" pod "ff44385a-0c7b-48c2-b43d-a83e007ad3ae" (UID: "ff44385a-0c7b-48c2-b43d-a83e007ad3ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:45.170941 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170879 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:45.170941 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.170913 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:45.174723 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.174680 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-token" pod "ff44385a-0c7b-48c2-b43d-a83e007ad3ae" (UID: "ff44385a-0c7b-48c2-b43d-a83e007ad3ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:45.174940 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.174918 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-envoy" pod "ff44385a-0c7b-48c2-b43d-a83e007ad3ae" (UID: "ff44385a-0c7b-48c2-b43d-a83e007ad3ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:45.175518 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.175494 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-podinfo" pod "ff44385a-0c7b-48c2-b43d-a83e007ad3ae" (UID: "ff44385a-0c7b-48c2-b43d-a83e007ad3ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:16:45.176646 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.176608 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-kube-api-access-82q79" pod "ff44385a-0c7b-48c2-b43d-a83e007ad3ae" (UID: "ff44385a-0c7b-48c2-b43d-a83e007ad3ae"). InnerVolumeSpecName "kube-api-access-82q79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:45.272033 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.271994 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:45.272033 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.272030 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82q79\" (UniqueName: \"kubernetes.io/projected/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-kube-api-access-82q79\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:45.272318 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.272044 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:45.272318 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.272057 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:45.272318 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:45.272068 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff44385a-0c7b-48c2-b43d-a83e007ad3ae-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:46.021851 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:46.021820 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" Oct 31 22:16:46.022344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:46.021852 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn" event={"ID":"ff44385a-0c7b-48c2-b43d-a83e007ad3ae","Type":"ContainerDied","Data":"64d4b2c315d99ebb82f4b8a5b8d9f19e7036a0419c97fbcdd960e601cbdcb483"} Oct 31 22:16:46.022344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:46.021930 5188 scope.go:122] "RemoveContainer" containerID="ed8c56ab5935075e85067f22a19f21a47911e7e3a18de4172d17f2ed79241c48" Oct 31 22:16:46.069566 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:46.069525 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn"] Oct 31 22:16:46.073737 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:46.073693 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-kr6rn"] Oct 31 22:16:47.474344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:47.474287 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" path="/var/lib/kubelet/pods/ff44385a-0c7b-48c2-b43d-a83e007ad3ae/volumes" Oct 31 22:16:49.053424 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.052455 5188 generic.go:356] "Generic (PLEG): container finished" podID="5a331fa0-7891-4369-9f99-861558258d5f" containerID="0a1605db271b840aa02af7bbaee57d467b7f30469dbebca2d3591123a41d5cf6" exitCode=0 Oct 31 22:16:49.053424 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.052533 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" event={"ID":"5a331fa0-7891-4369-9f99-861558258d5f","Type":"ContainerDied","Data":"0a1605db271b840aa02af7bbaee57d467b7f30469dbebca2d3591123a41d5cf6"} Oct 31 22:16:49.154907 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.154874 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:16:49.225097 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225062 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-istio-token\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-istio-token\") pod \"5a331fa0-7891-4369-9f99-861558258d5f\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " Oct 31 22:16:49.225305 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225161 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-certs\") pod \"5a331fa0-7891-4369-9f99-861558258d5f\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " Oct 31 22:16:49.225305 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225287 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-kube-api-access-c6hg4\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-kube-api-access-c6hg4\") pod \"5a331fa0-7891-4369-9f99-861558258d5f\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " Oct 31 22:16:49.225462 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225370 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a331fa0-7891-4369-9f99-861558258d5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a331fa0-7891-4369-9f99-861558258d5f-istiod-ca-cert\") pod \"5a331fa0-7891-4369-9f99-861558258d5f\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " Oct 31 22:16:49.225462 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225427 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-credential-socket\") pod \"5a331fa0-7891-4369-9f99-861558258d5f\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " Oct 31 22:16:49.225595 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225564 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-certs" pod "5a331fa0-7891-4369-9f99-861558258d5f" (UID: "5a331fa0-7891-4369-9f99-861558258d5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:49.225661 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225578 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-socket\") pod \"5a331fa0-7891-4369-9f99-861558258d5f\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " Oct 31 22:16:49.225721 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225672 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a331fa0-7891-4369-9f99-861558258d5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a331fa0-7891-4369-9f99-861558258d5f-istio-podinfo\") pod \"5a331fa0-7891-4369-9f99-861558258d5f\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " Oct 31 22:16:49.225721 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225679 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-credential-socket" pod "5a331fa0-7891-4369-9f99-861558258d5f" (UID: "5a331fa0-7891-4369-9f99-861558258d5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:49.225814 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225745 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-envoy\") pod \"5a331fa0-7891-4369-9f99-861558258d5f\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " Oct 31 22:16:49.225814 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.225779 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-data\") pod \"5a331fa0-7891-4369-9f99-861558258d5f\" (UID: \"5a331fa0-7891-4369-9f99-861558258d5f\") " Oct 31 22:16:49.226074 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.226054 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-data" pod "5a331fa0-7891-4369-9f99-861558258d5f" (UID: "5a331fa0-7891-4369-9f99-861558258d5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:49.226306 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.226289 5188 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.226425 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.226317 5188 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-credential-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.226425 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.226369 5188 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-data\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.226425 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.226286 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a331fa0-7891-4369-9f99-861558258d5f-istiod-ca-cert" pod "5a331fa0-7891-4369-9f99-861558258d5f" (UID: "5a331fa0-7891-4369-9f99-861558258d5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:49.226425 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.226354 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-socket" pod "5a331fa0-7891-4369-9f99-861558258d5f" (UID: "5a331fa0-7891-4369-9f99-861558258d5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:49.228545 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.228515 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-envoy" pod "5a331fa0-7891-4369-9f99-861558258d5f" (UID: "5a331fa0-7891-4369-9f99-861558258d5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:49.228718 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.228691 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-istio-token" pod "5a331fa0-7891-4369-9f99-861558258d5f" (UID: "5a331fa0-7891-4369-9f99-861558258d5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:49.228821 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.228801 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-kube-api-access-c6hg4" pod "5a331fa0-7891-4369-9f99-861558258d5f" (UID: "5a331fa0-7891-4369-9f99-861558258d5f"). InnerVolumeSpecName "kube-api-access-c6hg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:49.229105 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.229079 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a331fa0-7891-4369-9f99-861558258d5f-istio-podinfo" pod "5a331fa0-7891-4369-9f99-861558258d5f" (UID: "5a331fa0-7891-4369-9f99-861558258d5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:16:49.258518 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.258491 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 22:16:49.327467 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.327430 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/931df1cf-f6d0-4c1e-b87f-e596aba68266-kube-api-access-vvgq9\" (UniqueName: \"kubernetes.io/projected/931df1cf-f6d0-4c1e-b87f-e596aba68266-kube-api-access-vvgq9\") pod \"931df1cf-f6d0-4c1e-b87f-e596aba68266\" (UID: \"931df1cf-f6d0-4c1e-b87f-e596aba68266\") " Oct 31 22:16:49.327680 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.327504 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/931df1cf-f6d0-4c1e-b87f-e596aba68266-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931df1cf-f6d0-4c1e-b87f-e596aba68266-plugins-config-volume\") pod \"931df1cf-f6d0-4c1e-b87f-e596aba68266\" (UID: \"931df1cf-f6d0-4c1e-b87f-e596aba68266\") " Oct 31 22:16:49.327975 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.327940 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/931df1cf-f6d0-4c1e-b87f-e596aba68266-plugins-config-volume" pod "931df1cf-f6d0-4c1e-b87f-e596aba68266" (UID: "931df1cf-f6d0-4c1e-b87f-e596aba68266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:49.328107 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.328034 5188 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-istio-envoy\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.328107 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.328056 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.328107 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.328070 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6hg4\" (UniqueName: \"kubernetes.io/projected/5a331fa0-7891-4369-9f99-861558258d5f-kube-api-access-c6hg4\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.328107 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.328082 5188 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a331fa0-7891-4369-9f99-861558258d5f-istiod-ca-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.328107 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.328095 5188 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a331fa0-7891-4369-9f99-861558258d5f-workload-socket\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.328107 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.328107 5188 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a331fa0-7891-4369-9f99-861558258d5f-istio-podinfo\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.329833 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.329804 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/931df1cf-f6d0-4c1e-b87f-e596aba68266-kube-api-access-vvgq9" pod "931df1cf-f6d0-4c1e-b87f-e596aba68266" (UID: "931df1cf-f6d0-4c1e-b87f-e596aba68266"). InnerVolumeSpecName "kube-api-access-vvgq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:49.429234 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.429139 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvgq9\" (UniqueName: \"kubernetes.io/projected/931df1cf-f6d0-4c1e-b87f-e596aba68266-kube-api-access-vvgq9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.429234 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:49.429174 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/931df1cf-f6d0-4c1e-b87f-e596aba68266-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:50.060872 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.060835 5188 generic.go:356] "Generic (PLEG): container finished" podID="931df1cf-f6d0-4c1e-b87f-e596aba68266" containerID="169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697" exitCode=0 Oct 31 22:16:50.061434 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.060890 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" Oct 31 22:16:50.061434 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.060925 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" event={"ID":"931df1cf-f6d0-4c1e-b87f-e596aba68266","Type":"ContainerDied","Data":"169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697"} Oct 31 22:16:50.061434 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.060998 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm" event={"ID":"931df1cf-f6d0-4c1e-b87f-e596aba68266","Type":"ContainerDied","Data":"2deec949dd04c484f65c9c1f53912ab3e5c20e32a09eb33365792e9055ddbdbd"} Oct 31 22:16:50.061434 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.061033 5188 scope.go:122] "RemoveContainer" containerID="169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697" Oct 31 22:16:50.062610 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.062586 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" Oct 31 22:16:50.062736 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.062611 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9" event={"ID":"5a331fa0-7891-4369-9f99-861558258d5f","Type":"ContainerDied","Data":"0ea69a9a7d5b5333ca59320a957a4e0525ecc7c5df0a6c40bbfff9a74f31d6f8"} Oct 31 22:16:50.072471 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.072441 5188 scope.go:122] "RemoveContainer" containerID="169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697" Oct 31 22:16:50.072827 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:16:50.072801 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697\": container with ID starting with 169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697 not found: ID does not exist" containerID="169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697" Oct 31 22:16:50.072937 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.072842 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697"} err="failed to get container status \"169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697\": rpc error: code = NotFound desc = could not find container \"169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697\": container with ID starting with 169897b9a13c42422339b6b0458510839aa060337936fcc1d19068ec2120c697 not found: ID does not exist" Oct 31 22:16:50.072937 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.072875 5188 scope.go:122] "RemoveContainer" containerID="0a1605db271b840aa02af7bbaee57d467b7f30469dbebca2d3591123a41d5cf6" Oct 31 22:16:50.109053 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.109012 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm"] Oct 31 22:16:50.120317 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.120231 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-44405d-7c56f4d867-29mnm"] Oct 31 22:16:50.150801 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.150748 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9"] Oct 31 22:16:50.160261 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:50.160228 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-vxrb9"] Oct 31 22:16:51.473687 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:51.473639 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a331fa0-7891-4369-9f99-861558258d5f" path="/var/lib/kubelet/pods/5a331fa0-7891-4369-9f99-861558258d5f/volumes" Oct 31 22:16:51.474191 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:51.474082 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="931df1cf-f6d0-4c1e-b87f-e596aba68266" path="/var/lib/kubelet/pods/931df1cf-f6d0-4c1e-b87f-e596aba68266/volumes" Oct 31 22:16:59.110561 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.110516 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc"] Oct 31 22:16:59.111081 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.110814 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" podUID="e66030d3-d520-4c7c-b136-9ecdcdb6fd16" containerName="maas-api" containerID="cri-o://be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05" gracePeriod=30 Oct 31 22:16:59.227261 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.227214 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" podUID="2cb608c4-b046-4592-b919-6a26f69f6828" containerName="main" containerID="cri-o://6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6" gracePeriod=45 Oct 31 22:16:59.408407 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.405995 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 22:16:59.526954 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.526925 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:16:59.560689 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.560641 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-openshift-service-ca\") pod \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " Oct 31 22:16:59.560883 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.560856 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-maas-api-tls\") pod \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " Oct 31 22:16:59.560981 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.560964 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-kube-api-access-gf5fb\" (UniqueName: \"kubernetes.io/projected/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-kube-api-access-gf5fb\") pod \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " Oct 31 22:16:59.561023 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.561002 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-metrics-certs\") pod \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\" (UID: \"e66030d3-d520-4c7c-b136-9ecdcdb6fd16\") " Oct 31 22:16:59.561069 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.561011 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-openshift-service-ca" pod "e66030d3-d520-4c7c-b136-9ecdcdb6fd16" (UID: "e66030d3-d520-4c7c-b136-9ecdcdb6fd16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:59.561921 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.561629 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:59.563209 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.563183 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-maas-api-tls" pod "e66030d3-d520-4c7c-b136-9ecdcdb6fd16" (UID: "e66030d3-d520-4c7c-b136-9ecdcdb6fd16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:59.563379 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.563205 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-metrics-certs" pod "e66030d3-d520-4c7c-b136-9ecdcdb6fd16" (UID: "e66030d3-d520-4c7c-b136-9ecdcdb6fd16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:59.563379 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.563243 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-kube-api-access-gf5fb" pod "e66030d3-d520-4c7c-b136-9ecdcdb6fd16" (UID: "e66030d3-d520-4c7c-b136-9ecdcdb6fd16"). InnerVolumeSpecName "kube-api-access-gf5fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:59.662179 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.662077 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-home\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-home\") pod \"2cb608c4-b046-4592-b919-6a26f69f6828\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " Oct 31 22:16:59.662382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.662191 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb608c4-b046-4592-b919-6a26f69f6828-kube-api-access-bbx2s\" (UniqueName: \"kubernetes.io/projected/2cb608c4-b046-4592-b919-6a26f69f6828-kube-api-access-bbx2s\") pod \"2cb608c4-b046-4592-b919-6a26f69f6828\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " Oct 31 22:16:59.662382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.662227 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-dshm\") pod \"2cb608c4-b046-4592-b919-6a26f69f6828\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " Oct 31 22:16:59.662382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.662260 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb608c4-b046-4592-b919-6a26f69f6828-tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb608c4-b046-4592-b919-6a26f69f6828-tls-certs\") pod \"2cb608c4-b046-4592-b919-6a26f69f6828\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " Oct 31 22:16:59.662566 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.662392 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-model-cache\") pod \"2cb608c4-b046-4592-b919-6a26f69f6828\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " Oct 31 22:16:59.662566 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.662492 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-home" pod "2cb608c4-b046-4592-b919-6a26f69f6828" (UID: "2cb608c4-b046-4592-b919-6a26f69f6828"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:59.662566 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.662511 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-tmp-dir\") pod \"2cb608c4-b046-4592-b919-6a26f69f6828\" (UID: \"2cb608c4-b046-4592-b919-6a26f69f6828\") " Oct 31 22:16:59.662747 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.662685 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-model-cache" pod "2cb608c4-b046-4592-b919-6a26f69f6828" (UID: "2cb608c4-b046-4592-b919-6a26f69f6828"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:59.662808 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.662743 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-tmp-dir" pod "2cb608c4-b046-4592-b919-6a26f69f6828" (UID: "2cb608c4-b046-4592-b919-6a26f69f6828"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:59.663139 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.663102 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5fb\" (UniqueName: \"kubernetes.io/projected/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-kube-api-access-gf5fb\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:59.663139 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.663138 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:59.663139 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.663147 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:59.663139 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.663155 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:59.663469 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.663174 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e66030d3-d520-4c7c-b136-9ecdcdb6fd16-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:59.663469 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.663186 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:59.664852 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.664820 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb608c4-b046-4592-b919-6a26f69f6828-kube-api-access-bbx2s" pod "2cb608c4-b046-4592-b919-6a26f69f6828" (UID: "2cb608c4-b046-4592-b919-6a26f69f6828"). InnerVolumeSpecName "kube-api-access-bbx2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:59.664973 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.664862 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-dshm" pod "2cb608c4-b046-4592-b919-6a26f69f6828" (UID: "2cb608c4-b046-4592-b919-6a26f69f6828"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:59.665159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.665134 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb608c4-b046-4592-b919-6a26f69f6828-tls-certs" pod "2cb608c4-b046-4592-b919-6a26f69f6828" (UID: "2cb608c4-b046-4592-b919-6a26f69f6828"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:59.764901 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.764857 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbx2s\" (UniqueName: \"kubernetes.io/projected/2cb608c4-b046-4592-b919-6a26f69f6828-kube-api-access-bbx2s\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:59.764901 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.764890 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cb608c4-b046-4592-b919-6a26f69f6828-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:16:59.764901 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:16:59.764905 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cb608c4-b046-4592-b919-6a26f69f6828-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:00.161013 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.160974 5188 generic.go:356] "Generic (PLEG): container finished" podID="e66030d3-d520-4c7c-b136-9ecdcdb6fd16" containerID="be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05" exitCode=0 Oct 31 22:17:00.161547 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.161050 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" Oct 31 22:17:00.161547 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.161055 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" event={"ID":"e66030d3-d520-4c7c-b136-9ecdcdb6fd16","Type":"ContainerDied","Data":"be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05"} Oct 31 22:17:00.161547 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.161187 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc" event={"ID":"e66030d3-d520-4c7c-b136-9ecdcdb6fd16","Type":"ContainerDied","Data":"7afa622b9a2c4e17e4cb479856dc5884b0ba6e66d356d6c4ef5b40912247105f"} Oct 31 22:17:00.161547 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.161216 5188 scope.go:122] "RemoveContainer" containerID="be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05" Oct 31 22:17:00.163119 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.163091 5188 generic.go:356] "Generic (PLEG): container finished" podID="2cb608c4-b046-4592-b919-6a26f69f6828" containerID="6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6" exitCode=0 Oct 31 22:17:00.163275 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.163160 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" Oct 31 22:17:00.163275 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.163181 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" event={"ID":"2cb608c4-b046-4592-b919-6a26f69f6828","Type":"ContainerDied","Data":"6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6"} Oct 31 22:17:00.163275 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.163255 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2" event={"ID":"2cb608c4-b046-4592-b919-6a26f69f6828","Type":"ContainerDied","Data":"d0369dd473448714e1207345099b214ef9286376b7ad96d6fa6c1820b349e9d5"} Oct 31 22:17:00.174535 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.174514 5188 scope.go:122] "RemoveContainer" containerID="be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05" Oct 31 22:17:00.174821 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:17:00.174797 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05\": container with ID starting with be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05 not found: ID does not exist" containerID="be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05" Oct 31 22:17:00.174917 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.174827 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05"} err="failed to get container status \"be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05\": rpc error: code = NotFound desc = could not find container \"be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05\": container with ID starting with be4afa66eb5dc3f386c3e7afe0008a3e6199953403be5718fb86c9f5d0bd0b05 not found: ID does not exist" Oct 31 22:17:00.174917 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.174845 5188 scope.go:122] "RemoveContainer" containerID="6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6" Oct 31 22:17:00.186185 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.186159 5188 scope.go:122] "RemoveContainer" containerID="6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6" Oct 31 22:17:00.186542 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:17:00.186511 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6\": container with ID starting with 6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6 not found: ID does not exist" containerID="6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6" Oct 31 22:17:00.186659 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.186551 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6"} err="failed to get container status \"6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6\": rpc error: code = NotFound desc = could not find container \"6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6\": container with ID starting with 6ca2490418afbe259985b29759388ee41b23ad72af728a3a5bb84c5c4e1c24f6 not found: ID does not exist" Oct 31 22:17:00.206780 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.206743 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2"] Oct 31 22:17:00.216217 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.216172 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cd421c/test-model-cd421c-kserve-65f97f47f7-4f5q2"] Oct 31 22:17:00.241064 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.241019 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc"] Oct 31 22:17:00.245135 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:00.245094 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-44405d-b4d6cd76c-6qlvc"] Oct 31 22:17:01.370366 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.369813 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd"] Oct 31 22:17:01.370366 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.370065 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" podUID="2ab1c8d8-de2b-4020-bde4-d190e8494c7f" containerName="maas-api" containerID="cri-o://6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212" gracePeriod=30 Oct 31 22:17:01.473382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.473340 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb608c4-b046-4592-b919-6a26f69f6828" path="/var/lib/kubelet/pods/2cb608c4-b046-4592-b919-6a26f69f6828/volumes" Oct 31 22:17:01.474049 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.474023 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66030d3-d520-4c7c-b136-9ecdcdb6fd16" path="/var/lib/kubelet/pods/e66030d3-d520-4c7c-b136-9ecdcdb6fd16/volumes" Oct 31 22:17:01.671631 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.671601 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 22:17:01.795475 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.795430 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-kube-api-access-crlzj\" (UniqueName: \"kubernetes.io/projected/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-kube-api-access-crlzj\") pod \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " Oct 31 22:17:01.795721 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.795532 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-openshift-service-ca\") pod \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " Oct 31 22:17:01.795721 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.795641 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-metrics-certs\") pod \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " Oct 31 22:17:01.795857 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.795768 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-maas-api-tls\") pod \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\" (UID: \"2ab1c8d8-de2b-4020-bde4-d190e8494c7f\") " Oct 31 22:17:01.796068 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.795985 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-openshift-service-ca" pod "2ab1c8d8-de2b-4020-bde4-d190e8494c7f" (UID: "2ab1c8d8-de2b-4020-bde4-d190e8494c7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:01.796288 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.796255 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:01.798667 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.798627 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-metrics-certs" pod "2ab1c8d8-de2b-4020-bde4-d190e8494c7f" (UID: "2ab1c8d8-de2b-4020-bde4-d190e8494c7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:01.799010 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.798982 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-kube-api-access-crlzj" pod "2ab1c8d8-de2b-4020-bde4-d190e8494c7f" (UID: "2ab1c8d8-de2b-4020-bde4-d190e8494c7f"). InnerVolumeSpecName "kube-api-access-crlzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:01.799630 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.799604 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-maas-api-tls" pod "2ab1c8d8-de2b-4020-bde4-d190e8494c7f" (UID: "2ab1c8d8-de2b-4020-bde4-d190e8494c7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:01.897815 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.897715 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crlzj\" (UniqueName: \"kubernetes.io/projected/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-kube-api-access-crlzj\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:01.897815 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.897760 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:01.897815 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:01.897776 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab1c8d8-de2b-4020-bde4-d190e8494c7f-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:02.184835 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:02.184729 5188 generic.go:356] "Generic (PLEG): container finished" podID="2ab1c8d8-de2b-4020-bde4-d190e8494c7f" containerID="6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212" exitCode=0 Oct 31 22:17:02.184835 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:02.184772 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" Oct 31 22:17:02.184835 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:02.184810 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" event={"ID":"2ab1c8d8-de2b-4020-bde4-d190e8494c7f","Type":"ContainerDied","Data":"6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212"} Oct 31 22:17:02.185150 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:02.184882 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd" event={"ID":"2ab1c8d8-de2b-4020-bde4-d190e8494c7f","Type":"ContainerDied","Data":"8683b25367607c7244c8e11ea74cf2024ef1a6bd3ea91ffb982078c7f36da8e6"} Oct 31 22:17:02.185150 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:02.184914 5188 scope.go:122] "RemoveContainer" containerID="6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212" Oct 31 22:17:02.198153 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:02.198129 5188 scope.go:122] "RemoveContainer" containerID="6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212" Oct 31 22:17:02.198564 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:17:02.198534 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212\": container with ID starting with 6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212 not found: ID does not exist" containerID="6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212" Oct 31 22:17:02.198670 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:02.198579 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212"} err="failed to get container status \"6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212\": rpc error: code = NotFound desc = could not find container \"6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212\": container with ID starting with 6907d8927f07be4973a0d3a24967454327d5f0cd926aceda049a848f28dc5212 not found: ID does not exist" Oct 31 22:17:02.234350 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:02.234297 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd"] Oct 31 22:17:02.238255 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:02.238225 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-4bd51b-6966dcd677-ztswd"] Oct 31 22:17:02.540700 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:17:02.540615 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:17:03.474932 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:03.474888 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab1c8d8-de2b-4020-bde4-d190e8494c7f" path="/var/lib/kubelet/pods/2ab1c8d8-de2b-4020-bde4-d190e8494c7f/volumes" Oct 31 22:17:08.452651 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.452588 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9"] Oct 31 22:17:08.453371 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.452872 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" podUID="0f930b93-d00a-4c4f-a3be-bc32841ca19e" containerName="discovery" containerID="cri-o://d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a" gracePeriod=30 Oct 31 22:17:08.751029 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.750995 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 22:17:08.886974 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.886920 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f930b93-d00a-4c4f-a3be-bc32841ca19e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f930b93-d00a-4c4f-a3be-bc32841ca19e-local-certs\") pod \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " Oct 31 22:17:08.887199 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.887064 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-kube-api-access-2pjxk\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-kube-api-access-2pjxk\") pod \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " Oct 31 22:17:08.887199 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.887141 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-kubeconfig\") pod \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " Oct 31 22:17:08.887352 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.887202 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-dns-cert\") pod \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " Oct 31 22:17:08.887352 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.887283 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-ca-configmap\") pod \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " Oct 31 22:17:08.887352 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.887314 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-cacerts\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-cacerts\") pod \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " Oct 31 22:17:08.887530 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.887393 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-token\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-token\") pod \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\" (UID: \"0f930b93-d00a-4c4f-a3be-bc32841ca19e\") " Oct 31 22:17:08.888646 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.888609 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-ca-configmap" pod "0f930b93-d00a-4c4f-a3be-bc32841ca19e" (UID: "0f930b93-d00a-4c4f-a3be-bc32841ca19e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:08.893266 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.892685 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-dns-cert" pod "0f930b93-d00a-4c4f-a3be-bc32841ca19e" (UID: "0f930b93-d00a-4c4f-a3be-bc32841ca19e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:08.893266 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.892762 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-cacerts" pod "0f930b93-d00a-4c4f-a3be-bc32841ca19e" (UID: "0f930b93-d00a-4c4f-a3be-bc32841ca19e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:08.893266 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.892810 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f930b93-d00a-4c4f-a3be-bc32841ca19e-local-certs" pod "0f930b93-d00a-4c4f-a3be-bc32841ca19e" (UID: "0f930b93-d00a-4c4f-a3be-bc32841ca19e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:08.893266 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.892845 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-kubeconfig" pod "0f930b93-d00a-4c4f-a3be-bc32841ca19e" (UID: "0f930b93-d00a-4c4f-a3be-bc32841ca19e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:08.893266 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.892902 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-kube-api-access-2pjxk" pod "0f930b93-d00a-4c4f-a3be-bc32841ca19e" (UID: "0f930b93-d00a-4c4f-a3be-bc32841ca19e"). InnerVolumeSpecName "kube-api-access-2pjxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:08.893266 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.893218 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-token" pod "0f930b93-d00a-4c4f-a3be-bc32841ca19e" (UID: "0f930b93-d00a-4c4f-a3be-bc32841ca19e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:08.989193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.989058 5188 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-ca-configmap\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:08.989193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.989099 5188 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-cacerts\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:08.989193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.989116 5188 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-token\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:08.989193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.989130 5188 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f930b93-d00a-4c4f-a3be-bc32841ca19e-local-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:08.989193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.989145 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pjxk\" (UniqueName: \"kubernetes.io/projected/0f930b93-d00a-4c4f-a3be-bc32841ca19e-kube-api-access-2pjxk\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:08.989193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.989160 5188 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-kubeconfig\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:08.989193 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:08.989173 5188 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f930b93-d00a-4c4f-a3be-bc32841ca19e-istio-csr-dns-cert\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:09.247712 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.247622 5188 generic.go:356] "Generic (PLEG): container finished" podID="0f930b93-d00a-4c4f-a3be-bc32841ca19e" containerID="d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a" exitCode=0 Oct 31 22:17:09.247712 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.247666 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" Oct 31 22:17:09.247990 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.247714 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" event={"ID":"0f930b93-d00a-4c4f-a3be-bc32841ca19e","Type":"ContainerDied","Data":"d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a"} Oct 31 22:17:09.247990 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.247767 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9" event={"ID":"0f930b93-d00a-4c4f-a3be-bc32841ca19e","Type":"ContainerDied","Data":"7579a503bacea5ad3f7908cf36ef0b602e117ea6a1dcecef094b03e9cde3f6f3"} Oct 31 22:17:09.247990 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.247797 5188 scope.go:122] "RemoveContainer" containerID="d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a" Oct 31 22:17:09.264526 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.264503 5188 scope.go:122] "RemoveContainer" containerID="d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a" Oct 31 22:17:09.264825 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:17:09.264803 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a\": container with ID starting with d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a not found: ID does not exist" containerID="d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a" Oct 31 22:17:09.264890 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.264840 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a"} err="failed to get container status \"d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a\": rpc error: code = NotFound desc = could not find container \"d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a\": container with ID starting with d50749ecb1e11da4c374d41d13a19c6fbdc8a280e12b5d854c04403235b06d0a not found: ID does not exist" Oct 31 22:17:09.295857 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.295808 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9"] Oct 31 22:17:09.304766 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.304730 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r4hv9"] Oct 31 22:17:09.473547 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:09.473489 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f930b93-d00a-4c4f-a3be-bc32841ca19e" path="/var/lib/kubelet/pods/0f930b93-d00a-4c4f-a3be-bc32841ca19e/volumes" Oct 31 22:17:23.893117 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:23.893075 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5"] Oct 31 22:17:23.893692 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:23.893408 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" podUID="a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" containerName="maas-api" containerID="cri-o://4a7141e85dbf1fdba535bde2548193d88e2cec8cde6dad1210d505c3a04b1100" gracePeriod=30 Oct 31 22:17:24.356631 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.356591 5188 generic.go:356] "Generic (PLEG): container finished" podID="a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" containerID="4a7141e85dbf1fdba535bde2548193d88e2cec8cde6dad1210d505c3a04b1100" exitCode=0 Oct 31 22:17:24.356880 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.356860 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" event={"ID":"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2","Type":"ContainerDied","Data":"4a7141e85dbf1fdba535bde2548193d88e2cec8cde6dad1210d505c3a04b1100"} Oct 31 22:17:24.387981 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.387952 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:17:24.451260 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.451166 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-kube-api-access-wr7ks\" (UniqueName: \"kubernetes.io/projected/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-kube-api-access-wr7ks\") pod \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " Oct 31 22:17:24.451260 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.451226 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-openshift-service-ca\") pod \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " Oct 31 22:17:24.451621 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.451338 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-maas-api-tls\") pod \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " Oct 31 22:17:24.451621 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.451393 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-metrics-certs\") pod \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\" (UID: \"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2\") " Oct 31 22:17:24.451741 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.451677 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-openshift-service-ca" pod "a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" (UID: "a0b91cc9-b3f0-4fff-b1f3-049664e40ea2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:24.451861 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.451842 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:24.453514 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.453489 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-kube-api-access-wr7ks" pod "a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" (UID: "a0b91cc9-b3f0-4fff-b1f3-049664e40ea2"). InnerVolumeSpecName "kube-api-access-wr7ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:24.453669 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.453649 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-maas-api-tls" pod "a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" (UID: "a0b91cc9-b3f0-4fff-b1f3-049664e40ea2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:24.454011 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.453987 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-metrics-certs" pod "a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" (UID: "a0b91cc9-b3f0-4fff-b1f3-049664e40ea2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:24.553189 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.553140 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7ks\" (UniqueName: \"kubernetes.io/projected/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-kube-api-access-wr7ks\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:24.553189 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.553180 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:24.553189 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:24.553189 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:25.365000 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:25.364967 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" Oct 31 22:17:25.365559 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:25.365000 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5" event={"ID":"a0b91cc9-b3f0-4fff-b1f3-049664e40ea2","Type":"ContainerDied","Data":"c19ccc5348fefab793647b578148eac7bbdb271d1793afb02bf1dbfed0a6dc40"} Oct 31 22:17:25.365559 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:25.365065 5188 scope.go:122] "RemoveContainer" containerID="4a7141e85dbf1fdba535bde2548193d88e2cec8cde6dad1210d505c3a04b1100" Oct 31 22:17:25.411872 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:25.411826 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5"] Oct 31 22:17:25.422141 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:25.422104 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-cd421c-75f8c8cc96-xczf5"] Oct 31 22:17:25.473794 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:25.473756 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" path="/var/lib/kubelet/pods/a0b91cc9-b3f0-4fff-b1f3-049664e40ea2/volumes" Oct 31 22:17:32.540025 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:17:32.539978 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:17:57.575218 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.575097 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2"] Oct 31 22:17:57.594222 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.594174 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" podUID="a039b42c-216c-433c-9a1c-a200e9ab5374" containerName="maas-api" containerID="cri-o://96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a" gracePeriod=30 Oct 31 22:17:57.873392 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.873358 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:17:57.929841 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.929804 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a039b42c-216c-433c-9a1c-a200e9ab5374-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a039b42c-216c-433c-9a1c-a200e9ab5374-openshift-service-ca\") pod \"a039b42c-216c-433c-9a1c-a200e9ab5374\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " Oct 31 22:17:57.930060 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.929853 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-metrics-certs\") pod \"a039b42c-216c-433c-9a1c-a200e9ab5374\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " Oct 31 22:17:57.930060 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.929943 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls\") pod \"a039b42c-216c-433c-9a1c-a200e9ab5374\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " Oct 31 22:17:57.930205 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.930067 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a039b42c-216c-433c-9a1c-a200e9ab5374-kube-api-access-2m6lf\" (UniqueName: \"kubernetes.io/projected/a039b42c-216c-433c-9a1c-a200e9ab5374-kube-api-access-2m6lf\") pod \"a039b42c-216c-433c-9a1c-a200e9ab5374\" (UID: \"a039b42c-216c-433c-9a1c-a200e9ab5374\") " Oct 31 22:17:57.930271 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.930228 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a039b42c-216c-433c-9a1c-a200e9ab5374-openshift-service-ca" pod "a039b42c-216c-433c-9a1c-a200e9ab5374" (UID: "a039b42c-216c-433c-9a1c-a200e9ab5374"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:57.930536 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.930513 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a039b42c-216c-433c-9a1c-a200e9ab5374-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:57.932174 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.932149 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-metrics-certs" pod "a039b42c-216c-433c-9a1c-a200e9ab5374" (UID: "a039b42c-216c-433c-9a1c-a200e9ab5374"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:57.932664 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.932639 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls" pod "a039b42c-216c-433c-9a1c-a200e9ab5374" (UID: "a039b42c-216c-433c-9a1c-a200e9ab5374"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:57.932780 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:57.932747 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a039b42c-216c-433c-9a1c-a200e9ab5374-kube-api-access-2m6lf" pod "a039b42c-216c-433c-9a1c-a200e9ab5374" (UID: "a039b42c-216c-433c-9a1c-a200e9ab5374"). InnerVolumeSpecName "kube-api-access-2m6lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:58.031694 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.031658 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m6lf\" (UniqueName: \"kubernetes.io/projected/a039b42c-216c-433c-9a1c-a200e9ab5374-kube-api-access-2m6lf\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:58.031694 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.031690 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:58.031694 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.031700 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a039b42c-216c-433c-9a1c-a200e9ab5374-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:17:58.602278 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.602241 5188 generic.go:356] "Generic (PLEG): container finished" podID="a039b42c-216c-433c-9a1c-a200e9ab5374" containerID="96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a" exitCode=0 Oct 31 22:17:58.602753 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.602384 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" Oct 31 22:17:58.602753 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.602392 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" event={"ID":"a039b42c-216c-433c-9a1c-a200e9ab5374","Type":"ContainerDied","Data":"96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a"} Oct 31 22:17:58.602753 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.602468 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2" event={"ID":"a039b42c-216c-433c-9a1c-a200e9ab5374","Type":"ContainerDied","Data":"3ca63bec088ce7ed2bbd3d9ee866eec9aa1291659ef22bfbcfe17591ea0d3022"} Oct 31 22:17:58.602753 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.602499 5188 scope.go:122] "RemoveContainer" containerID="96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a" Oct 31 22:17:58.617688 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.617662 5188 scope.go:122] "RemoveContainer" containerID="96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a" Oct 31 22:17:58.617979 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:17:58.617953 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a\": container with ID starting with 96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a not found: ID does not exist" containerID="96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a" Oct 31 22:17:58.618086 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.617986 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a"} err="failed to get container status \"96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a\": rpc error: code = NotFound desc = could not find container \"96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a\": container with ID starting with 96fac9f9a5f6374ffd07761f6dd512113dacc87d9ada3e61f60fc5115726ed4a not found: ID does not exist" Oct 31 22:17:58.648226 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.648189 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2"] Oct 31 22:17:58.657567 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:58.657535 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c08360-6548dbf6bb-n8jm2"] Oct 31 22:17:59.473870 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:17:59.473824 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a039b42c-216c-433c-9a1c-a200e9ab5374" path="/var/lib/kubelet/pods/a039b42c-216c-433c-9a1c-a200e9ab5374/volumes" Oct 31 22:18:02.537827 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:18:02.537786 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:18:31.395158 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.395047 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5"] Oct 31 22:18:31.395899 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.395315 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" podUID="9428ee1e-11af-405f-a792-0971229ef18c" containerName="maas-api" containerID="cri-o://0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792" gracePeriod=30 Oct 31 22:18:31.674372 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.674338 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 22:18:31.712469 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.712279 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9428ee1e-11af-405f-a792-0971229ef18c-kube-api-access-wwm2q\" (UniqueName: \"kubernetes.io/projected/9428ee1e-11af-405f-a792-0971229ef18c-kube-api-access-wwm2q\") pod \"9428ee1e-11af-405f-a792-0971229ef18c\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " Oct 31 22:18:31.713077 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.713038 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9428ee1e-11af-405f-a792-0971229ef18c-openshift-service-ca" pod "9428ee1e-11af-405f-a792-0971229ef18c" (UID: "9428ee1e-11af-405f-a792-0971229ef18c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:31.713239 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.713120 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9428ee1e-11af-405f-a792-0971229ef18c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9428ee1e-11af-405f-a792-0971229ef18c-openshift-service-ca\") pod \"9428ee1e-11af-405f-a792-0971229ef18c\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " Oct 31 22:18:31.713312 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.713252 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-maas-api-tls\") pod \"9428ee1e-11af-405f-a792-0971229ef18c\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " Oct 31 22:18:31.713408 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.713308 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-metrics-certs\") pod \"9428ee1e-11af-405f-a792-0971229ef18c\" (UID: \"9428ee1e-11af-405f-a792-0971229ef18c\") " Oct 31 22:18:31.713938 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.713908 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9428ee1e-11af-405f-a792-0971229ef18c-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:31.716456 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.716415 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9428ee1e-11af-405f-a792-0971229ef18c-kube-api-access-wwm2q" pod "9428ee1e-11af-405f-a792-0971229ef18c" (UID: "9428ee1e-11af-405f-a792-0971229ef18c"). InnerVolumeSpecName "kube-api-access-wwm2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:31.716456 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.716420 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-maas-api-tls" pod "9428ee1e-11af-405f-a792-0971229ef18c" (UID: "9428ee1e-11af-405f-a792-0971229ef18c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:31.717087 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.717070 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-metrics-certs" pod "9428ee1e-11af-405f-a792-0971229ef18c" (UID: "9428ee1e-11af-405f-a792-0971229ef18c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:31.814975 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.814923 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:31.814975 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.814967 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9428ee1e-11af-405f-a792-0971229ef18c-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:31.815266 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.814985 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwm2q\" (UniqueName: \"kubernetes.io/projected/9428ee1e-11af-405f-a792-0971229ef18c-kube-api-access-wwm2q\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:31.840609 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.840560 5188 generic.go:356] "Generic (PLEG): container finished" podID="9428ee1e-11af-405f-a792-0971229ef18c" containerID="0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792" exitCode=0 Oct 31 22:18:31.840828 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.840647 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" Oct 31 22:18:31.840828 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.840715 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" event={"ID":"9428ee1e-11af-405f-a792-0971229ef18c","Type":"ContainerDied","Data":"0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792"} Oct 31 22:18:31.840828 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.840758 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5" event={"ID":"9428ee1e-11af-405f-a792-0971229ef18c","Type":"ContainerDied","Data":"7ffd0ec198840d405f2dbe6251cc5838a22decd2e970e7116578c027fcfddb22"} Oct 31 22:18:31.840828 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.840779 5188 scope.go:122] "RemoveContainer" containerID="0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792" Oct 31 22:18:31.856560 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.856527 5188 scope.go:122] "RemoveContainer" containerID="0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792" Oct 31 22:18:31.856982 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:18:31.856935 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792\": container with ID starting with 0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792 not found: ID does not exist" containerID="0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792" Oct 31 22:18:31.857113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.856978 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792"} err="failed to get container status \"0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792\": rpc error: code = NotFound desc = could not find container \"0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792\": container with ID starting with 0778ee35f9ac994336790920d16531c32e470913cdefc2f8129a0fff39210792 not found: ID does not exist" Oct 31 22:18:31.890283 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.890245 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5"] Oct 31 22:18:31.894689 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:31.894646 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-696f9a-b4ddd6cb4-7lwt5"] Oct 31 22:18:32.538527 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:18:32.538492 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:18:33.473392 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:33.473348 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9428ee1e-11af-405f-a792-0971229ef18c" path="/var/lib/kubelet/pods/9428ee1e-11af-405f-a792-0971229ef18c/volumes" Oct 31 22:18:50.017451 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.017407 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b"] Oct 31 22:18:50.018000 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.017692 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" podUID="11611511-4c92-4cd5-81a1-c15b30598446" containerName="maas-api" containerID="cri-o://50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730" gracePeriod=30 Oct 31 22:18:50.287989 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.287920 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 22:18:50.425992 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.425947 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-maas-api-tls\") pod \"11611511-4c92-4cd5-81a1-c15b30598446\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " Oct 31 22:18:50.426196 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.426042 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-metrics-certs\") pod \"11611511-4c92-4cd5-81a1-c15b30598446\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " Oct 31 22:18:50.426196 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.426094 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11611511-4c92-4cd5-81a1-c15b30598446-kube-api-access-56hg6\" (UniqueName: \"kubernetes.io/projected/11611511-4c92-4cd5-81a1-c15b30598446-kube-api-access-56hg6\") pod \"11611511-4c92-4cd5-81a1-c15b30598446\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " Oct 31 22:18:50.426357 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.426249 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11611511-4c92-4cd5-81a1-c15b30598446-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11611511-4c92-4cd5-81a1-c15b30598446-openshift-service-ca\") pod \"11611511-4c92-4cd5-81a1-c15b30598446\" (UID: \"11611511-4c92-4cd5-81a1-c15b30598446\") " Oct 31 22:18:50.426665 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.426629 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11611511-4c92-4cd5-81a1-c15b30598446-openshift-service-ca" pod "11611511-4c92-4cd5-81a1-c15b30598446" (UID: "11611511-4c92-4cd5-81a1-c15b30598446"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:50.426842 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.426824 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11611511-4c92-4cd5-81a1-c15b30598446-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:50.428275 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.428249 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11611511-4c92-4cd5-81a1-c15b30598446-kube-api-access-56hg6" pod "11611511-4c92-4cd5-81a1-c15b30598446" (UID: "11611511-4c92-4cd5-81a1-c15b30598446"). InnerVolumeSpecName "kube-api-access-56hg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:50.428806 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.428777 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-metrics-certs" pod "11611511-4c92-4cd5-81a1-c15b30598446" (UID: "11611511-4c92-4cd5-81a1-c15b30598446"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:50.428911 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.428819 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-maas-api-tls" pod "11611511-4c92-4cd5-81a1-c15b30598446" (UID: "11611511-4c92-4cd5-81a1-c15b30598446"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:50.528535 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.528493 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56hg6\" (UniqueName: \"kubernetes.io/projected/11611511-4c92-4cd5-81a1-c15b30598446-kube-api-access-56hg6\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:50.528535 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.528537 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:50.528752 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.528552 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11611511-4c92-4cd5-81a1-c15b30598446-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:50.972181 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.972146 5188 generic.go:356] "Generic (PLEG): container finished" podID="11611511-4c92-4cd5-81a1-c15b30598446" containerID="50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730" exitCode=0 Oct 31 22:18:50.972417 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.972206 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" Oct 31 22:18:50.972417 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.972227 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" event={"ID":"11611511-4c92-4cd5-81a1-c15b30598446","Type":"ContainerDied","Data":"50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730"} Oct 31 22:18:50.972417 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.972294 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b" event={"ID":"11611511-4c92-4cd5-81a1-c15b30598446","Type":"ContainerDied","Data":"11b26386ceb5a73e2f02ede0d41a8ef881ea1d3adbe0c816cdf5d1f93aa41708"} Oct 31 22:18:50.972417 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.972342 5188 scope.go:122] "RemoveContainer" containerID="50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730" Oct 31 22:18:50.984859 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.984827 5188 scope.go:122] "RemoveContainer" containerID="50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730" Oct 31 22:18:50.985218 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:18:50.985194 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730\": container with ID starting with 50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730 not found: ID does not exist" containerID="50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730" Oct 31 22:18:50.985334 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:50.985226 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730"} err="failed to get container status \"50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730\": rpc error: code = NotFound desc = could not find container \"50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730\": container with ID starting with 50e2e59b6943245b216a1d9635a8bc16730bd89b17643c4c18f4c16f3c6ca730 not found: ID does not exist" Oct 31 22:18:51.019231 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:51.019179 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b"] Oct 31 22:18:51.030648 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:51.030603 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-c744c0-75fbdcdd5-fp58b"] Oct 31 22:18:51.473411 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:51.473370 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11611511-4c92-4cd5-81a1-c15b30598446" path="/var/lib/kubelet/pods/11611511-4c92-4cd5-81a1-c15b30598446/volumes" Oct 31 22:18:55.477267 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:55.477232 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm"] Oct 31 22:18:55.479359 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:55.479304 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" podUID="906b0063-d7e6-4434-861f-051bef5e818f" containerName="maas-api" containerID="cri-o://dc1a25d8c3a1a9aa0ce8aa2c2752456a3e1c2a9c41799b0a9fa8d6f126f983ab" gracePeriod=30 Oct 31 22:18:55.802960 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:55.802926 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh"] Oct 31 22:18:55.803158 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:55.803131 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" podUID="3a248b6d-2243-4d70-92c4-73dec22d0d7d" containerName="maas-api" containerID="cri-o://b73a520d08f1eb87af86ad7a09d468062e996f2bd7fbd6bef82eede0afc046f1" gracePeriod=30 Oct 31 22:18:56.023155 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.023119 5188 generic.go:356] "Generic (PLEG): container finished" podID="906b0063-d7e6-4434-861f-051bef5e818f" containerID="dc1a25d8c3a1a9aa0ce8aa2c2752456a3e1c2a9c41799b0a9fa8d6f126f983ab" exitCode=0 Oct 31 22:18:56.023382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.023197 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" event={"ID":"906b0063-d7e6-4434-861f-051bef5e818f","Type":"ContainerDied","Data":"dc1a25d8c3a1a9aa0ce8aa2c2752456a3e1c2a9c41799b0a9fa8d6f126f983ab"} Oct 31 22:18:56.023382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.023267 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" event={"ID":"906b0063-d7e6-4434-861f-051bef5e818f","Type":"ContainerDied","Data":"5713b4be864b70b3a01d67e5e3f4b17dfa7f651d5424c31679aa542b0fe4dd95"} Oct 31 22:18:56.023382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.023289 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5713b4be864b70b3a01d67e5e3f4b17dfa7f651d5424c31679aa542b0fe4dd95" Oct 31 22:18:56.025449 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.025418 5188 generic.go:356] "Generic (PLEG): container finished" podID="3a248b6d-2243-4d70-92c4-73dec22d0d7d" containerID="b73a520d08f1eb87af86ad7a09d468062e996f2bd7fbd6bef82eede0afc046f1" exitCode=0 Oct 31 22:18:56.025590 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.025504 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" event={"ID":"3a248b6d-2243-4d70-92c4-73dec22d0d7d","Type":"ContainerDied","Data":"b73a520d08f1eb87af86ad7a09d468062e996f2bd7fbd6bef82eede0afc046f1"} Oct 31 22:18:56.026404 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.026362 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:18:56.088733 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.088691 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-maas-api-tls\") pod \"906b0063-d7e6-4434-861f-051bef5e818f\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " Oct 31 22:18:56.089023 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.088770 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-metrics-certs\") pod \"906b0063-d7e6-4434-861f-051bef5e818f\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " Oct 31 22:18:56.089023 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.088814 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906b0063-d7e6-4434-861f-051bef5e818f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0063-d7e6-4434-861f-051bef5e818f-openshift-service-ca\") pod \"906b0063-d7e6-4434-861f-051bef5e818f\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " Oct 31 22:18:56.089023 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.088864 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906b0063-d7e6-4434-861f-051bef5e818f-kube-api-access-glhwf\" (UniqueName: \"kubernetes.io/projected/906b0063-d7e6-4434-861f-051bef5e818f-kube-api-access-glhwf\") pod \"906b0063-d7e6-4434-861f-051bef5e818f\" (UID: \"906b0063-d7e6-4434-861f-051bef5e818f\") " Oct 31 22:18:56.090259 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.090206 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906b0063-d7e6-4434-861f-051bef5e818f-openshift-service-ca" pod "906b0063-d7e6-4434-861f-051bef5e818f" (UID: "906b0063-d7e6-4434-861f-051bef5e818f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:56.092181 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.092141 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906b0063-d7e6-4434-861f-051bef5e818f-kube-api-access-glhwf" pod "906b0063-d7e6-4434-861f-051bef5e818f" (UID: "906b0063-d7e6-4434-861f-051bef5e818f"). InnerVolumeSpecName "kube-api-access-glhwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:56.095972 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.095836 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-metrics-certs" pod "906b0063-d7e6-4434-861f-051bef5e818f" (UID: "906b0063-d7e6-4434-861f-051bef5e818f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:56.095972 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.095932 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-maas-api-tls" pod "906b0063-d7e6-4434-861f-051bef5e818f" (UID: "906b0063-d7e6-4434-861f-051bef5e818f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:56.118461 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.118428 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 22:18:56.190056 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.190013 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-metrics-certs\") pod \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " Oct 31 22:18:56.190604 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.190579 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls\") pod \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " Oct 31 22:18:56.191006 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.190982 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a248b6d-2243-4d70-92c4-73dec22d0d7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a248b6d-2243-4d70-92c4-73dec22d0d7d-openshift-service-ca\") pod \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " Oct 31 22:18:56.191428 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.191402 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a248b6d-2243-4d70-92c4-73dec22d0d7d-openshift-service-ca" pod "3a248b6d-2243-4d70-92c4-73dec22d0d7d" (UID: "3a248b6d-2243-4d70-92c4-73dec22d0d7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:56.191516 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.191487 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a248b6d-2243-4d70-92c4-73dec22d0d7d-kube-api-access-vfncs\" (UniqueName: \"kubernetes.io/projected/3a248b6d-2243-4d70-92c4-73dec22d0d7d-kube-api-access-vfncs\") pod \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\" (UID: \"3a248b6d-2243-4d70-92c4-73dec22d0d7d\") " Oct 31 22:18:56.192260 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.192234 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a248b6d-2243-4d70-92c4-73dec22d0d7d-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:56.192372 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.192266 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:56.192372 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.192281 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906b0063-d7e6-4434-861f-051bef5e818f-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:56.192372 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.192295 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906b0063-d7e6-4434-861f-051bef5e818f-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:56.192372 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.192309 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhwf\" (UniqueName: \"kubernetes.io/projected/906b0063-d7e6-4434-861f-051bef5e818f-kube-api-access-glhwf\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:56.194932 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.194885 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls" pod "3a248b6d-2243-4d70-92c4-73dec22d0d7d" (UID: "3a248b6d-2243-4d70-92c4-73dec22d0d7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:56.197331 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.197287 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-metrics-certs" pod "3a248b6d-2243-4d70-92c4-73dec22d0d7d" (UID: "3a248b6d-2243-4d70-92c4-73dec22d0d7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:56.202722 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.202693 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a248b6d-2243-4d70-92c4-73dec22d0d7d-kube-api-access-vfncs" pod "3a248b6d-2243-4d70-92c4-73dec22d0d7d" (UID: "3a248b6d-2243-4d70-92c4-73dec22d0d7d"). InnerVolumeSpecName "kube-api-access-vfncs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:56.293850 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.293750 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:56.293850 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.293789 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a248b6d-2243-4d70-92c4-73dec22d0d7d-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:56.293850 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:56.293802 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfncs\" (UniqueName: \"kubernetes.io/projected/3a248b6d-2243-4d70-92c4-73dec22d0d7d-kube-api-access-vfncs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:18:57.033896 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.033786 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" event={"ID":"3a248b6d-2243-4d70-92c4-73dec22d0d7d","Type":"ContainerDied","Data":"89731e9f700328a2f21b1bee99f907cd6287ec7d2ce8e0630f118f89af35c3d0"} Oct 31 22:18:57.033896 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.033874 5188 scope.go:122] "RemoveContainer" containerID="b73a520d08f1eb87af86ad7a09d468062e996f2bd7fbd6bef82eede0afc046f1" Oct 31 22:18:57.034499 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.033806 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm" Oct 31 22:18:57.034499 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.033821 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh" Oct 31 22:18:57.086896 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.086851 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm"] Oct 31 22:18:57.099968 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.099926 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1b0cce-68774d6b8c-cq7jm"] Oct 31 22:18:57.129666 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.129626 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh"] Oct 31 22:18:57.132302 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.132270 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4a73b1-6b7555688d-kcdgh"] Oct 31 22:18:57.359033 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.358946 5188 scope.go:122] "RemoveContainer" containerID="dc1a25d8c3a1a9aa0ce8aa2c2752456a3e1c2a9c41799b0a9fa8d6f126f983ab" Oct 31 22:18:57.474294 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.474257 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a248b6d-2243-4d70-92c4-73dec22d0d7d" path="/var/lib/kubelet/pods/3a248b6d-2243-4d70-92c4-73dec22d0d7d/volumes" Oct 31 22:18:57.474745 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:18:57.474730 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906b0063-d7e6-4434-861f-051bef5e818f" path="/var/lib/kubelet/pods/906b0063-d7e6-4434-861f-051bef5e818f/volumes" Oct 31 22:19:02.538459 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:19:02.538422 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:19:03.619609 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:03.619559 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v"] Oct 31 22:19:09.362238 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:09.362196 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9"] Oct 31 22:19:14.363277 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:14.363217 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" podUID="145a082c-6551-481b-864f-703dacdcd332" containerName="payload-processing" containerID="cri-o://ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18" gracePeriod=25 Oct 31 22:19:14.957104 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:14.957059 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq"] Oct 31 22:19:15.746929 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:15.746891 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh"] Oct 31 22:19:17.082778 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.082739 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l"] Oct 31 22:19:17.083231 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.082960 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" podUID="1ab743db-44f2-455e-ba49-4d0afea44c71" containerName="maas-api" containerID="cri-o://7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b" gracePeriod=30 Oct 31 22:19:17.376475 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.376446 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 22:19:17.530722 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.530684 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-maas-api-tls\") pod \"1ab743db-44f2-455e-ba49-4d0afea44c71\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " Oct 31 22:19:17.530960 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.530745 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab743db-44f2-455e-ba49-4d0afea44c71-kube-api-access-hwpk5\" (UniqueName: \"kubernetes.io/projected/1ab743db-44f2-455e-ba49-4d0afea44c71-kube-api-access-hwpk5\") pod \"1ab743db-44f2-455e-ba49-4d0afea44c71\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " Oct 31 22:19:17.530960 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.530764 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-metrics-certs\") pod \"1ab743db-44f2-455e-ba49-4d0afea44c71\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " Oct 31 22:19:17.530960 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.530792 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab743db-44f2-455e-ba49-4d0afea44c71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab743db-44f2-455e-ba49-4d0afea44c71-openshift-service-ca\") pod \"1ab743db-44f2-455e-ba49-4d0afea44c71\" (UID: \"1ab743db-44f2-455e-ba49-4d0afea44c71\") " Oct 31 22:19:17.531402 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.531358 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab743db-44f2-455e-ba49-4d0afea44c71-openshift-service-ca" pod "1ab743db-44f2-455e-ba49-4d0afea44c71" (UID: "1ab743db-44f2-455e-ba49-4d0afea44c71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:17.533196 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.533171 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab743db-44f2-455e-ba49-4d0afea44c71-kube-api-access-hwpk5" pod "1ab743db-44f2-455e-ba49-4d0afea44c71" (UID: "1ab743db-44f2-455e-ba49-4d0afea44c71"). InnerVolumeSpecName "kube-api-access-hwpk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:17.533290 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.533211 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-metrics-certs" pod "1ab743db-44f2-455e-ba49-4d0afea44c71" (UID: "1ab743db-44f2-455e-ba49-4d0afea44c71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:17.533290 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.533228 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-maas-api-tls" pod "1ab743db-44f2-455e-ba49-4d0afea44c71" (UID: "1ab743db-44f2-455e-ba49-4d0afea44c71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:17.632845 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.632801 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:17.632845 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.632834 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpk5\" (UniqueName: \"kubernetes.io/projected/1ab743db-44f2-455e-ba49-4d0afea44c71-kube-api-access-hwpk5\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:17.632845 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.632845 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab743db-44f2-455e-ba49-4d0afea44c71-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:17.633113 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:17.632854 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab743db-44f2-455e-ba49-4d0afea44c71-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:18.175077 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.175040 5188 generic.go:356] "Generic (PLEG): container finished" podID="1ab743db-44f2-455e-ba49-4d0afea44c71" containerID="7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b" exitCode=0 Oct 31 22:19:18.175660 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.175091 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" Oct 31 22:19:18.175660 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.175125 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" event={"ID":"1ab743db-44f2-455e-ba49-4d0afea44c71","Type":"ContainerDied","Data":"7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b"} Oct 31 22:19:18.175660 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.175187 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l" event={"ID":"1ab743db-44f2-455e-ba49-4d0afea44c71","Type":"ContainerDied","Data":"14c0b988ea95aa56d3b4517ba12e3f6cb183c618280b4dfffebec40c0f7666a9"} Oct 31 22:19:18.175660 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.175216 5188 scope.go:122] "RemoveContainer" containerID="7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b" Oct 31 22:19:18.187938 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.187909 5188 scope.go:122] "RemoveContainer" containerID="7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b" Oct 31 22:19:18.188304 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:19:18.188280 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b\": container with ID starting with 7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b not found: ID does not exist" containerID="7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b" Oct 31 22:19:18.188440 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.188337 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b"} err="failed to get container status \"7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b\": rpc error: code = NotFound desc = could not find container \"7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b\": container with ID starting with 7c5fd41be12a0ca287156a608e17a39f5506075f499e57eb01dcdea7be94710b not found: ID does not exist" Oct 31 22:19:18.223956 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.223907 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l"] Oct 31 22:19:18.231814 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.231776 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-b36e62-5fc865b5c8-9798l"] Oct 31 22:19:18.641755 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.641709 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" containerID="cri-o://7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38" gracePeriod=45 Oct 31 22:19:18.917574 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:18.917535 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 22:19:19.047857 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.047816 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4b299a-da61-44bd-ae6f-5da0499400c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4b299a-da61-44bd-ae6f-5da0499400c8-tls-certs\") pod \"3f4b299a-da61-44bd-ae6f-5da0499400c8\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " Oct 31 22:19:19.048059 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.047940 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-home\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-home\") pod \"3f4b299a-da61-44bd-ae6f-5da0499400c8\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " Oct 31 22:19:19.048059 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.047976 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4b299a-da61-44bd-ae6f-5da0499400c8-kube-api-access-djl94\" (UniqueName: \"kubernetes.io/projected/3f4b299a-da61-44bd-ae6f-5da0499400c8-kube-api-access-djl94\") pod \"3f4b299a-da61-44bd-ae6f-5da0499400c8\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " Oct 31 22:19:19.048191 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.048095 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-dshm\") pod \"3f4b299a-da61-44bd-ae6f-5da0499400c8\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " Oct 31 22:19:19.048191 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.048128 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-model-cache\") pod \"3f4b299a-da61-44bd-ae6f-5da0499400c8\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " Oct 31 22:19:19.048319 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.048234 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-home" pod "3f4b299a-da61-44bd-ae6f-5da0499400c8" (UID: "3f4b299a-da61-44bd-ae6f-5da0499400c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:19.048461 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.048437 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-model-cache" pod "3f4b299a-da61-44bd-ae6f-5da0499400c8" (UID: "3f4b299a-da61-44bd-ae6f-5da0499400c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:19.048538 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.048519 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-tmp-dir\") pod \"3f4b299a-da61-44bd-ae6f-5da0499400c8\" (UID: \"3f4b299a-da61-44bd-ae6f-5da0499400c8\") " Oct 31 22:19:19.048601 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.048542 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-tmp-dir" pod "3f4b299a-da61-44bd-ae6f-5da0499400c8" (UID: "3f4b299a-da61-44bd-ae6f-5da0499400c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:19.048907 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.048889 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.049024 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.048933 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.049024 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.048946 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.050541 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.050507 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-dshm" pod "3f4b299a-da61-44bd-ae6f-5da0499400c8" (UID: "3f4b299a-da61-44bd-ae6f-5da0499400c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:19.050658 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.050621 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4b299a-da61-44bd-ae6f-5da0499400c8-tls-certs" pod "3f4b299a-da61-44bd-ae6f-5da0499400c8" (UID: "3f4b299a-da61-44bd-ae6f-5da0499400c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:19.050717 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.050688 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4b299a-da61-44bd-ae6f-5da0499400c8-kube-api-access-djl94" pod "3f4b299a-da61-44bd-ae6f-5da0499400c8" (UID: "3f4b299a-da61-44bd-ae6f-5da0499400c8"). InnerVolumeSpecName "kube-api-access-djl94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:19.149830 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.149728 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4b299a-da61-44bd-ae6f-5da0499400c8-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.149830 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.149769 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4b299a-da61-44bd-ae6f-5da0499400c8-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.149830 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.149782 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djl94\" (UniqueName: \"kubernetes.io/projected/3f4b299a-da61-44bd-ae6f-5da0499400c8-kube-api-access-djl94\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.187514 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.187469 5188 generic.go:356] "Generic (PLEG): container finished" podID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerID="7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38" exitCode=0 Oct 31 22:19:19.187985 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.187547 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" event={"ID":"3f4b299a-da61-44bd-ae6f-5da0499400c8","Type":"ContainerDied","Data":"7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38"} Oct 31 22:19:19.187985 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.187601 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" event={"ID":"3f4b299a-da61-44bd-ae6f-5da0499400c8","Type":"ContainerDied","Data":"cd90046403a856420d4de7a1b7a1dda5746bf794b2144ffc4bd5a3b1d76b9755"} Oct 31 22:19:19.187985 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.187558 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v" Oct 31 22:19:19.187985 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.187623 5188 scope.go:122] "RemoveContainer" containerID="7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38" Oct 31 22:19:19.199143 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.199119 5188 scope.go:122] "RemoveContainer" containerID="7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38" Oct 31 22:19:19.199519 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:19:19.199499 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38\": container with ID starting with 7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38 not found: ID does not exist" containerID="7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38" Oct 31 22:19:19.199566 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.199531 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38"} err="failed to get container status \"7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38\": rpc error: code = NotFound desc = could not find container \"7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38\": container with ID starting with 7dd28948f76a1ddd0f885dc6a3e9e1b2b5e3bb9c6041b61cab3e7ae72265eb38 not found: ID does not exist" Oct 31 22:19:19.232720 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.232680 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v"] Oct 31 22:19:19.242363 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.242312 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/simulator-w4-cc45a7-kserve-68b458c76d-2nr8v"] Oct 31 22:19:19.476205 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.476112 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab743db-44f2-455e-ba49-4d0afea44c71" path="/var/lib/kubelet/pods/1ab743db-44f2-455e-ba49-4d0afea44c71/volumes" Oct 31 22:19:19.476815 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.476791 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" path="/var/lib/kubelet/pods/3f4b299a-da61-44bd-ae6f-5da0499400c8/volumes" Oct 31 22:19:19.958054 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:19.958007 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" podUID="52ab05c6-b656-41d0-982b-3c5b0e8e07c4" containerName="payload-processing" containerID="cri-o://c27edaf9cf97225aff1dc7358922ea37d6bfc26edf3c45e1910e565fd5aafd94" gracePeriod=25 Oct 31 22:19:20.202162 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.201874 5188 generic.go:356] "Generic (PLEG): container finished" podID="52ab05c6-b656-41d0-982b-3c5b0e8e07c4" containerID="c27edaf9cf97225aff1dc7358922ea37d6bfc26edf3c45e1910e565fd5aafd94" exitCode=0 Oct 31 22:19:20.202162 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.201959 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" event={"ID":"52ab05c6-b656-41d0-982b-3c5b0e8e07c4","Type":"ContainerDied","Data":"c27edaf9cf97225aff1dc7358922ea37d6bfc26edf3c45e1910e565fd5aafd94"} Oct 31 22:19:20.233616 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.233579 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:19:20.363495 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.363458 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-kube-api-access-65mnq\" (UniqueName: \"kubernetes.io/projected/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-kube-api-access-65mnq\") pod \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\" (UID: \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\") " Oct 31 22:19:20.363720 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.363548 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-plugins-config-volume\") pod \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\" (UID: \"52ab05c6-b656-41d0-982b-3c5b0e8e07c4\") " Oct 31 22:19:20.364009 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.363980 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-plugins-config-volume" pod "52ab05c6-b656-41d0-982b-3c5b0e8e07c4" (UID: "52ab05c6-b656-41d0-982b-3c5b0e8e07c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:20.365913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.365883 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-kube-api-access-65mnq" pod "52ab05c6-b656-41d0-982b-3c5b0e8e07c4" (UID: "52ab05c6-b656-41d0-982b-3c5b0e8e07c4"). InnerVolumeSpecName "kube-api-access-65mnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:20.464670 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.464577 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65mnq\" (UniqueName: \"kubernetes.io/projected/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-kube-api-access-65mnq\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:20.464670 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.464608 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab05c6-b656-41d0-982b-3c5b0e8e07c4-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:20.748044 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:20.747882 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" podUID="64815867-ef42-4773-99ab-b5669ffe1265" containerName="payload-processing" containerID="cri-o://84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28" gracePeriod=25 Oct 31 22:19:21.211506 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:21.211462 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" event={"ID":"52ab05c6-b656-41d0-982b-3c5b0e8e07c4","Type":"ContainerDied","Data":"282cec757327e8bf0619efea14455ecab47db07bc4721f002662a461c2a7976b"} Oct 31 22:19:21.211506 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:21.211491 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq" Oct 31 22:19:21.212109 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:21.211546 5188 scope.go:122] "RemoveContainer" containerID="c27edaf9cf97225aff1dc7358922ea37d6bfc26edf3c45e1910e565fd5aafd94" Oct 31 22:19:21.262768 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:21.262722 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq"] Oct 31 22:19:21.272350 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:21.272288 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-5c54fc-5f9f868c59-s5bgq"] Oct 31 22:19:21.472883 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:21.472786 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ab05c6-b656-41d0-982b-3c5b0e8e07c4" path="/var/lib/kubelet/pods/52ab05c6-b656-41d0-982b-3c5b0e8e07c4/volumes" Oct 31 22:19:22.415995 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.415953 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk"] Oct 31 22:19:22.416543 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.416186 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" podUID="f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" containerName="maas-api" containerID="cri-o://2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669" gracePeriod=30 Oct 31 22:19:22.691267 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.691238 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:19:22.786699 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.786657 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-openshift-service-ca\") pod \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " Oct 31 22:19:22.786941 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.786750 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-metrics-certs\") pod \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " Oct 31 22:19:22.786941 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.786801 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls\") pod \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " Oct 31 22:19:22.787068 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.786964 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-kube-api-access-6vbjq\" (UniqueName: \"kubernetes.io/projected/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-kube-api-access-6vbjq\") pod \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\" (UID: \"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c\") " Oct 31 22:19:22.787132 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.787110 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-openshift-service-ca" pod "f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" (UID: "f353bffb-dcb8-43c9-b3ee-dd33bd811d0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:22.787403 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.787352 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:22.789804 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.789772 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-metrics-certs" pod "f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" (UID: "f353bffb-dcb8-43c9-b3ee-dd33bd811d0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:22.789943 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.789844 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls" pod "f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" (UID: "f353bffb-dcb8-43c9-b3ee-dd33bd811d0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:22.789943 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.789907 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-kube-api-access-6vbjq" pod "f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" (UID: "f353bffb-dcb8-43c9-b3ee-dd33bd811d0c"). InnerVolumeSpecName "kube-api-access-6vbjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:22.880766 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.880718 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" podUID="64815867-ef42-4773-99ab-b5669ffe1265" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:19:22.888484 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.888407 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vbjq\" (UniqueName: \"kubernetes.io/projected/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-kube-api-access-6vbjq\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:22.888484 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.888445 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:22.888484 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:22.888459 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:23.227363 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.227306 5188 generic.go:356] "Generic (PLEG): container finished" podID="f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" containerID="2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669" exitCode=0 Oct 31 22:19:23.227586 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.227390 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" event={"ID":"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c","Type":"ContainerDied","Data":"2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669"} Oct 31 22:19:23.227586 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.227445 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" event={"ID":"f353bffb-dcb8-43c9-b3ee-dd33bd811d0c","Type":"ContainerDied","Data":"db2b4ec7dc26322b0c57b9f2bdb4fb9243bd490fa45840733a5016705784fbde"} Oct 31 22:19:23.227586 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.227471 5188 scope.go:122] "RemoveContainer" containerID="2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669" Oct 31 22:19:23.227586 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.227400 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk" Oct 31 22:19:23.238884 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.238864 5188 scope.go:122] "RemoveContainer" containerID="2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669" Oct 31 22:19:23.239212 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:19:23.239194 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669\": container with ID starting with 2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669 not found: ID does not exist" containerID="2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669" Oct 31 22:19:23.239252 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.239222 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669"} err="failed to get container status \"2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669\": rpc error: code = NotFound desc = could not find container \"2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669\": container with ID starting with 2109848f5130559493faddfd83cc07d505cab19cda354c6c3628f68566cdf669 not found: ID does not exist" Oct 31 22:19:23.273837 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.273800 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk"] Oct 31 22:19:23.277649 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.277619 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5c54fc-55f9885f56-wxqmk"] Oct 31 22:19:23.473053 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.473006 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" path="/var/lib/kubelet/pods/f353bffb-dcb8-43c9-b3ee-dd33bd811d0c/volumes" Oct 31 22:19:23.853974 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:23.853937 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" podUID="145a082c-6551-481b-864f-703dacdcd332" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:19:32.540309 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:19:32.540266 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:19:32.880305 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:32.880262 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" podUID="64815867-ef42-4773-99ab-b5669ffe1265" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:19:33.853749 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:33.853697 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" podUID="145a082c-6551-481b-864f-703dacdcd332" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:19:34.625466 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:34.625369 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 22:19:34.708854 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:34.708821 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/145a082c-6551-481b-864f-703dacdcd332-kube-api-access-25trr\" (UniqueName: \"kubernetes.io/projected/145a082c-6551-481b-864f-703dacdcd332-kube-api-access-25trr\") pod \"145a082c-6551-481b-864f-703dacdcd332\" (UID: \"145a082c-6551-481b-864f-703dacdcd332\") " Oct 31 22:19:34.709130 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:34.708888 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/145a082c-6551-481b-864f-703dacdcd332-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145a082c-6551-481b-864f-703dacdcd332-plugins-config-volume\") pod \"145a082c-6551-481b-864f-703dacdcd332\" (UID: \"145a082c-6551-481b-864f-703dacdcd332\") " Oct 31 22:19:34.709421 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:34.709398 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/145a082c-6551-481b-864f-703dacdcd332-plugins-config-volume" pod "145a082c-6551-481b-864f-703dacdcd332" (UID: "145a082c-6551-481b-864f-703dacdcd332"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:34.711849 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:34.711822 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/145a082c-6551-481b-864f-703dacdcd332-kube-api-access-25trr" pod "145a082c-6551-481b-864f-703dacdcd332" (UID: "145a082c-6551-481b-864f-703dacdcd332"). InnerVolumeSpecName "kube-api-access-25trr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:34.810067 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:34.810018 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25trr\" (UniqueName: \"kubernetes.io/projected/145a082c-6551-481b-864f-703dacdcd332-kube-api-access-25trr\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.810067 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:34.810057 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/145a082c-6551-481b-864f-703dacdcd332-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:35.307967 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.307921 5188 generic.go:356] "Generic (PLEG): container finished" podID="145a082c-6551-481b-864f-703dacdcd332" containerID="ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18" exitCode=0 Oct 31 22:19:35.308511 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.307970 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" Oct 31 22:19:35.308511 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.307977 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" event={"ID":"145a082c-6551-481b-864f-703dacdcd332","Type":"ContainerDied","Data":"ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18"} Oct 31 22:19:35.308511 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.308100 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9" event={"ID":"145a082c-6551-481b-864f-703dacdcd332","Type":"ContainerDied","Data":"6fbd22ee9b4a70db7425edbcaab7cd6099240cf4690ec4d320f4b6ee9775faa9"} Oct 31 22:19:35.308511 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.308130 5188 scope.go:122] "RemoveContainer" containerID="ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18" Oct 31 22:19:35.319246 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.319220 5188 scope.go:122] "RemoveContainer" containerID="ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18" Oct 31 22:19:35.319599 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:19:35.319561 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18\": container with ID starting with ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18 not found: ID does not exist" containerID="ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18" Oct 31 22:19:35.319742 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.319606 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18"} err="failed to get container status \"ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18\": rpc error: code = NotFound desc = could not find container \"ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18\": container with ID starting with ba71b52677b15f2d5e2c85e1590260597bbbebcce2ab3f2717749de661370b18 not found: ID does not exist" Oct 31 22:19:35.358027 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.357992 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9"] Oct 31 22:19:35.366628 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.366596 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-cc45a7-67fd74fc57-d2tc9"] Oct 31 22:19:35.472953 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:35.472918 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="145a082c-6551-481b-864f-703dacdcd332" path="/var/lib/kubelet/pods/145a082c-6551-481b-864f-703dacdcd332/volumes" Oct 31 22:19:38.352956 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:38.352917 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd"] Oct 31 22:19:41.018114 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.018036 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 22:19:41.080095 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.080042 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64815867-ef42-4773-99ab-b5669ffe1265-kube-api-access-k8g7s\" (UniqueName: \"kubernetes.io/projected/64815867-ef42-4773-99ab-b5669ffe1265-kube-api-access-k8g7s\") pod \"64815867-ef42-4773-99ab-b5669ffe1265\" (UID: \"64815867-ef42-4773-99ab-b5669ffe1265\") " Oct 31 22:19:41.080349 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.080123 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64815867-ef42-4773-99ab-b5669ffe1265-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64815867-ef42-4773-99ab-b5669ffe1265-plugins-config-volume\") pod \"64815867-ef42-4773-99ab-b5669ffe1265\" (UID: \"64815867-ef42-4773-99ab-b5669ffe1265\") " Oct 31 22:19:41.080587 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.080558 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64815867-ef42-4773-99ab-b5669ffe1265-plugins-config-volume" pod "64815867-ef42-4773-99ab-b5669ffe1265" (UID: "64815867-ef42-4773-99ab-b5669ffe1265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:41.082866 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.082825 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64815867-ef42-4773-99ab-b5669ffe1265-kube-api-access-k8g7s" pod "64815867-ef42-4773-99ab-b5669ffe1265" (UID: "64815867-ef42-4773-99ab-b5669ffe1265"). InnerVolumeSpecName "kube-api-access-k8g7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:41.181068 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.181024 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8g7s\" (UniqueName: \"kubernetes.io/projected/64815867-ef42-4773-99ab-b5669ffe1265-kube-api-access-k8g7s\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:41.181068 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.181059 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64815867-ef42-4773-99ab-b5669ffe1265-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:41.353272 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.353168 5188 generic.go:356] "Generic (PLEG): container finished" podID="64815867-ef42-4773-99ab-b5669ffe1265" containerID="84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28" exitCode=0 Oct 31 22:19:41.353272 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.353228 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" Oct 31 22:19:41.353272 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.353257 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" event={"ID":"64815867-ef42-4773-99ab-b5669ffe1265","Type":"ContainerDied","Data":"84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28"} Oct 31 22:19:41.353598 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.353317 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh" event={"ID":"64815867-ef42-4773-99ab-b5669ffe1265","Type":"ContainerDied","Data":"455ffca2e2307013e0c18148d4e3d592b7cfc102ad1c4a52a35c42e1975dc188"} Oct 31 22:19:41.353598 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.353363 5188 scope.go:122] "RemoveContainer" containerID="84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28" Oct 31 22:19:41.363663 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.363636 5188 scope.go:122] "RemoveContainer" containerID="84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28" Oct 31 22:19:41.363986 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:19:41.363960 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28\": container with ID starting with 84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28 not found: ID does not exist" containerID="84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28" Oct 31 22:19:41.364092 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.363994 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28"} err="failed to get container status \"84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28\": rpc error: code = NotFound desc = could not find container \"84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28\": container with ID starting with 84186d9b347a81fdbe26518d878b57fa7bd6192c0b8f69c13caa4fec47af1e28 not found: ID does not exist" Oct 31 22:19:41.401999 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.401955 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh"] Oct 31 22:19:41.406543 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.406505 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b36e62-5bb6746995-v2lgh"] Oct 31 22:19:41.473215 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.473168 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64815867-ef42-4773-99ab-b5669ffe1265" path="/var/lib/kubelet/pods/64815867-ef42-4773-99ab-b5669ffe1265/volumes" Oct 31 22:19:41.731742 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.731700 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n"] Oct 31 22:19:41.731983 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:41.731954 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" podUID="df656d96-07dd-4660-a2dc-810e24f43929" containerName="maas-api" containerID="cri-o://a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99" gracePeriod=30 Oct 31 22:19:42.009342 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.009301 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 22:19:42.091065 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.091028 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df656d96-07dd-4660-a2dc-810e24f43929-kube-api-access-zkj8c\" (UniqueName: \"kubernetes.io/projected/df656d96-07dd-4660-a2dc-810e24f43929-kube-api-access-zkj8c\") pod \"df656d96-07dd-4660-a2dc-810e24f43929\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " Oct 31 22:19:42.091644 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.091091 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-maas-api-tls\") pod \"df656d96-07dd-4660-a2dc-810e24f43929\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " Oct 31 22:19:42.091644 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.091187 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df656d96-07dd-4660-a2dc-810e24f43929-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df656d96-07dd-4660-a2dc-810e24f43929-openshift-service-ca\") pod \"df656d96-07dd-4660-a2dc-810e24f43929\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " Oct 31 22:19:42.091644 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.091223 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-metrics-certs\") pod \"df656d96-07dd-4660-a2dc-810e24f43929\" (UID: \"df656d96-07dd-4660-a2dc-810e24f43929\") " Oct 31 22:19:42.091841 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.091670 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df656d96-07dd-4660-a2dc-810e24f43929-openshift-service-ca" pod "df656d96-07dd-4660-a2dc-810e24f43929" (UID: "df656d96-07dd-4660-a2dc-810e24f43929"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:42.093810 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.093775 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-maas-api-tls" pod "df656d96-07dd-4660-a2dc-810e24f43929" (UID: "df656d96-07dd-4660-a2dc-810e24f43929"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:42.093954 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.093879 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-metrics-certs" pod "df656d96-07dd-4660-a2dc-810e24f43929" (UID: "df656d96-07dd-4660-a2dc-810e24f43929"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:42.093954 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.093930 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df656d96-07dd-4660-a2dc-810e24f43929-kube-api-access-zkj8c" pod "df656d96-07dd-4660-a2dc-810e24f43929" (UID: "df656d96-07dd-4660-a2dc-810e24f43929"). InnerVolumeSpecName "kube-api-access-zkj8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:42.192670 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.192632 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkj8c\" (UniqueName: \"kubernetes.io/projected/df656d96-07dd-4660-a2dc-810e24f43929-kube-api-access-zkj8c\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:42.192670 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.192665 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:42.192670 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.192675 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df656d96-07dd-4660-a2dc-810e24f43929-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:42.193017 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.192684 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df656d96-07dd-4660-a2dc-810e24f43929-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:19:42.361603 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.361573 5188 generic.go:356] "Generic (PLEG): container finished" podID="df656d96-07dd-4660-a2dc-810e24f43929" containerID="a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99" exitCode=0 Oct 31 22:19:42.361805 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.361628 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" Oct 31 22:19:42.361805 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.361649 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" event={"ID":"df656d96-07dd-4660-a2dc-810e24f43929","Type":"ContainerDied","Data":"a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99"} Oct 31 22:19:42.361805 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.361708 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n" event={"ID":"df656d96-07dd-4660-a2dc-810e24f43929","Type":"ContainerDied","Data":"1db8050c0882af2096d75a7a1e212535c86f698408918ab317e64ffebea7b86c"} Oct 31 22:19:42.361805 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.361725 5188 scope.go:122] "RemoveContainer" containerID="a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99" Oct 31 22:19:42.372814 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.372785 5188 scope.go:122] "RemoveContainer" containerID="a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99" Oct 31 22:19:42.373143 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:19:42.373120 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99\": container with ID starting with a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99 not found: ID does not exist" containerID="a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99" Oct 31 22:19:42.373215 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.373152 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99"} err="failed to get container status \"a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99\": rpc error: code = NotFound desc = could not find container \"a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99\": container with ID starting with a65617b8f5445c5e934f733a0b742177f1b393e25e8bf411538ea2406d2fcc99 not found: ID does not exist" Oct 31 22:19:42.407318 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.407278 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n"] Oct 31 22:19:42.412487 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:42.412450 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fb32d4-67886dcd4-bns8n"] Oct 31 22:19:43.353943 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:43.353889 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" podUID="eb07ecde-e243-402e-965e-be6a87cc6d6a" containerName="payload-pre-processing" containerID="cri-o://0c0ccb7736cb0d82a741e9d6c3d544725614156a894ba7bc2370d872978799ed" gracePeriod=25 Oct 31 22:19:43.472790 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:43.472751 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df656d96-07dd-4660-a2dc-810e24f43929" path="/var/lib/kubelet/pods/df656d96-07dd-4660-a2dc-810e24f43929/volumes" Oct 31 22:19:52.472636 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:19:52.472591 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" podUID="eb07ecde-e243-402e-965e-be6a87cc6d6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:20:01.020505 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.020467 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p"] Oct 31 22:20:01.021186 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021160 5188 cpu_manager.go:414] "containerMap: removing container" podUID="145a082c-6551-481b-864f-703dacdcd332" containerName="payload-processing" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021188 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="145a082c-6551-481b-864f-703dacdcd332" containerName="payload-processing" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021209 5188 cpu_manager.go:414] "containerMap: removing container" podUID="52ab05c6-b656-41d0-982b-3c5b0e8e07c4" containerName="payload-processing" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021219 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ab05c6-b656-41d0-982b-3c5b0e8e07c4" containerName="payload-processing" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021237 5188 cpu_manager.go:414] "containerMap: removing container" podUID="906b0063-d7e6-4434-861f-051bef5e818f" containerName="maas-api" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021244 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906b0063-d7e6-4434-861f-051bef5e818f" containerName="maas-api" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021255 5188 cpu_manager.go:414] "containerMap: removing container" podUID="df656d96-07dd-4660-a2dc-810e24f43929" containerName="maas-api" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021263 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df656d96-07dd-4660-a2dc-810e24f43929" containerName="maas-api" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021286 5188 cpu_manager.go:414] "containerMap: removing container" podUID="abb5e084-18e0-47d2-85f7-a5102d5a8055" containerName="maas-api" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021292 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb5e084-18e0-47d2-85f7-a5102d5a8055" containerName="maas-api" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021299 5188 cpu_manager.go:414] "containerMap: removing container" podUID="db87ebb5-bb07-4598-b6e0-c3d587a05cbf" containerName="main" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021303 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db87ebb5-bb07-4598-b6e0-c3d587a05cbf" containerName="main" Oct 31 22:20:01.021314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021310 5188 cpu_manager.go:414] "containerMap: removing container" podUID="e66030d3-d520-4c7c-b136-9ecdcdb6fd16" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021316 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66030d3-d520-4c7c-b136-9ecdcdb6fd16" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021358 5188 cpu_manager.go:414] "containerMap: removing container" podUID="1ab743db-44f2-455e-ba49-4d0afea44c71" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021364 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab743db-44f2-455e-ba49-4d0afea44c71" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021371 5188 cpu_manager.go:414] "containerMap: removing container" podUID="931df1cf-f6d0-4c1e-b87f-e596aba68266" containerName="payload-pre-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021376 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="931df1cf-f6d0-4c1e-b87f-e596aba68266" containerName="payload-pre-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021385 5188 cpu_manager.go:414] "containerMap: removing container" podUID="68f1815c-225d-4141-9279-eb765de32aa3" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021389 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f1815c-225d-4141-9279-eb765de32aa3" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021398 5188 cpu_manager.go:414] "containerMap: removing container" podUID="64815867-ef42-4773-99ab-b5669ffe1265" containerName="payload-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021404 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64815867-ef42-4773-99ab-b5669ffe1265" containerName="payload-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021416 5188 cpu_manager.go:414] "containerMap: removing container" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021425 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021434 5188 cpu_manager.go:414] "containerMap: removing container" podUID="11611511-4c92-4cd5-81a1-c15b30598446" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021439 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11611511-4c92-4cd5-81a1-c15b30598446" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021446 5188 cpu_manager.go:414] "containerMap: removing container" podUID="2ab1c8d8-de2b-4020-bde4-d190e8494c7f" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021451 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab1c8d8-de2b-4020-bde4-d190e8494c7f" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021456 5188 cpu_manager.go:414] "containerMap: removing container" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021460 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021467 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021471 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021477 5188 cpu_manager.go:414] "containerMap: removing container" podUID="0f930b93-d00a-4c4f-a3be-bc32841ca19e" containerName="discovery" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021481 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f930b93-d00a-4c4f-a3be-bc32841ca19e" containerName="discovery" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021487 5188 cpu_manager.go:414] "containerMap: removing container" podUID="a039b42c-216c-433c-9a1c-a200e9ab5374" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021491 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a039b42c-216c-433c-9a1c-a200e9ab5374" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021503 5188 cpu_manager.go:414] "containerMap: removing container" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021510 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021518 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021522 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021533 5188 cpu_manager.go:414] "containerMap: removing container" podUID="a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021537 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021543 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9428ee1e-11af-405f-a792-0971229ef18c" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021548 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9428ee1e-11af-405f-a792-0971229ef18c" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021556 5188 cpu_manager.go:414] "containerMap: removing container" podUID="da10e874-0602-4ca3-86aa-500bb5b32448" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021561 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da10e874-0602-4ca3-86aa-500bb5b32448" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021568 5188 cpu_manager.go:414] "containerMap: removing container" podUID="95a84925-746c-4331-8f5a-197e34cd274d" containerName="payload-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021572 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95a84925-746c-4331-8f5a-197e34cd274d" containerName="payload-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021580 5188 cpu_manager.go:414] "containerMap: removing container" podUID="2cb608c4-b046-4592-b919-6a26f69f6828" containerName="main" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021587 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb608c4-b046-4592-b919-6a26f69f6828" containerName="main" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021597 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021602 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021607 5188 cpu_manager.go:414] "containerMap: removing container" podUID="83e142d5-6162-47b4-9f68-c1c16aa8a778" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021612 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e142d5-6162-47b4-9f68-c1c16aa8a778" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021620 5188 cpu_manager.go:414] "containerMap: removing container" podUID="3a248b6d-2243-4d70-92c4-73dec22d0d7d" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021624 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a248b6d-2243-4d70-92c4-73dec22d0d7d" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021632 5188 cpu_manager.go:414] "containerMap: removing container" podUID="853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021636 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021645 5188 cpu_manager.go:414] "containerMap: removing container" podUID="f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021651 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021741 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="145a082c-6551-481b-864f-703dacdcd332" containerName="payload-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021752 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ab05c6-b656-41d0-982b-3c5b0e8e07c4" containerName="payload-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021759 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="906b0063-d7e6-4434-861f-051bef5e818f" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021768 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="df656d96-07dd-4660-a2dc-810e24f43929" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021777 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb5e084-18e0-47d2-85f7-a5102d5a8055" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021783 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="db87ebb5-bb07-4598-b6e0-c3d587a05cbf" containerName="main" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021790 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66030d3-d520-4c7c-b136-9ecdcdb6fd16" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021796 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab743db-44f2-455e-ba49-4d0afea44c71" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021801 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="931df1cf-f6d0-4c1e-b87f-e596aba68266" containerName="payload-pre-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021806 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f1815c-225d-4141-9279-eb765de32aa3" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021815 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="64815867-ef42-4773-99ab-b5669ffe1265" containerName="payload-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021824 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c89147-f74d-4df3-b722-48fbf418de11" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021829 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="11611511-4c92-4cd5-81a1-c15b30598446" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021834 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab1c8d8-de2b-4020-bde4-d190e8494c7f" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021838 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a019eb-d5f0-41f8-9822-eeb0c265e906" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021843 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a331fa0-7891-4369-9f99-861558258d5f" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021849 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f930b93-d00a-4c4f-a3be-bc32841ca19e" containerName="discovery" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021854 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="a039b42c-216c-433c-9a1c-a200e9ab5374" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021859 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4b299a-da61-44bd-ae6f-5da0499400c8" containerName="main" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021864 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd42e3ee-f036-4bc4-87ab-45103198c952" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021872 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b91cc9-b3f0-4fff-b1f3-049664e40ea2" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021878 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9428ee1e-11af-405f-a792-0971229ef18c" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021883 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="da10e874-0602-4ca3-86aa-500bb5b32448" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021889 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="95a84925-746c-4331-8f5a-197e34cd274d" containerName="payload-processing" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021899 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb608c4-b046-4592-b919-6a26f69f6828" containerName="main" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021906 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff44385a-0c7b-48c2-b43d-a83e007ad3ae" containerName="istio-proxy" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021912 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e142d5-6162-47b4-9f68-c1c16aa8a778" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021917 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a248b6d-2243-4d70-92c4-73dec22d0d7d" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021922 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="853d5e9a-0ff1-4dd9-be7c-efda8dfaa456" containerName="maas-api" Oct 31 22:20:01.022034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.021927 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="f353bffb-dcb8-43c9-b3ee-dd33bd811d0c" containerName="maas-api" Oct 31 22:20:01.026314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.024954 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.038069 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.038039 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:20:01.048456 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.048420 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:20:01.052295 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.052260 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p"] Oct 31 22:20:01.189317 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.189276 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b8f395-t6d9p\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.189521 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.189441 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b8f395-t6d9p\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.189579 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.189518 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkfj6\" (UniqueName: \"kubernetes.io/projected/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-kube-api-access-mkfj6\") pod \"maas-api-revoke-sub-d0b8f395-t6d9p\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.227636 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.227595 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt"] Oct 31 22:20:01.263150 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.263113 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz"] Oct 31 22:20:01.290351 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.290245 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkfj6\" (UniqueName: \"kubernetes.io/projected/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-kube-api-access-mkfj6\") pod \"maas-api-revoke-sub-d0b8f395-t6d9p\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.290351 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.290295 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b8f395-t6d9p\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.290583 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.290367 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b8f395-t6d9p\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.291254 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.291224 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0b8f395-t6d9p\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.293106 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.293074 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0b8f395-t6d9p\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.313803 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.313766 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkfj6\" (UniqueName: \"kubernetes.io/projected/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-kube-api-access-mkfj6\") pod \"maas-api-revoke-sub-d0b8f395-t6d9p\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.341576 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.341530 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:01.652683 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:01.652652 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p"] Oct 31 22:20:01.656902 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:20:01.656868 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff750fd3_bf7c_45a5_bbb2_04dd0d8f0392.slice/crio-13b479be04be80887701d03a7c4bd09da05cf77bb851a6dd21eca4c936a3f4a1 WatchSource:0}: Error finding container 13b479be04be80887701d03a7c4bd09da05cf77bb851a6dd21eca4c936a3f4a1: Status 404 returned error can't find the container with id 13b479be04be80887701d03a7c4bd09da05cf77bb851a6dd21eca4c936a3f4a1 Oct 31 22:20:02.472729 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:02.472681 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" podUID="eb07ecde-e243-402e-965e-be6a87cc6d6a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:20:02.497969 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:02.497927 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" event={"ID":"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392","Type":"ContainerStarted","Data":"51ce23d0e7506a138f16824ffd881acdd8dfaf21b0e28f4db855a08eca826cda"} Oct 31 22:20:02.498199 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:02.497988 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" event={"ID":"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392","Type":"ContainerStarted","Data":"13b479be04be80887701d03a7c4bd09da05cf77bb851a6dd21eca4c936a3f4a1"} Oct 31 22:20:02.538812 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:20:02.538768 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:20:02.545015 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:02.544959 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" podStartSLOduration=2.5449426859999997 podStartE2EDuration="2.544942686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:20:02.533370849 +0000 UTC m=+3849.226328243" watchObservedRunningTime="2026-10-31 22:20:02.544942686 +0000 UTC m=+3849.237900078" Oct 31 22:20:03.506599 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.506558 5188 generic.go:356] "Generic (PLEG): container finished" podID="ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392" containerID="51ce23d0e7506a138f16824ffd881acdd8dfaf21b0e28f4db855a08eca826cda" exitCode=0 Oct 31 22:20:03.507135 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.506644 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" event={"ID":"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392","Type":"ContainerDied","Data":"51ce23d0e7506a138f16824ffd881acdd8dfaf21b0e28f4db855a08eca826cda"} Oct 31 22:20:03.508552 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.508523 5188 generic.go:356] "Generic (PLEG): container finished" podID="eb07ecde-e243-402e-965e-be6a87cc6d6a" containerID="0c0ccb7736cb0d82a741e9d6c3d544725614156a894ba7bc2370d872978799ed" exitCode=0 Oct 31 22:20:03.508687 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.508573 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" event={"ID":"eb07ecde-e243-402e-965e-be6a87cc6d6a","Type":"ContainerDied","Data":"0c0ccb7736cb0d82a741e9d6c3d544725614156a894ba7bc2370d872978799ed"} Oct 31 22:20:03.649362 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.649315 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 22:20:03.721765 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.721720 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb07ecde-e243-402e-965e-be6a87cc6d6a-kube-api-access-46nkm\" (UniqueName: \"kubernetes.io/projected/eb07ecde-e243-402e-965e-be6a87cc6d6a-kube-api-access-46nkm\") pod \"eb07ecde-e243-402e-965e-be6a87cc6d6a\" (UID: \"eb07ecde-e243-402e-965e-be6a87cc6d6a\") " Oct 31 22:20:03.724009 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.723974 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb07ecde-e243-402e-965e-be6a87cc6d6a-kube-api-access-46nkm" pod "eb07ecde-e243-402e-965e-be6a87cc6d6a" (UID: "eb07ecde-e243-402e-965e-be6a87cc6d6a"). InnerVolumeSpecName "kube-api-access-46nkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:03.822640 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.822594 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb07ecde-e243-402e-965e-be6a87cc6d6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb07ecde-e243-402e-965e-be6a87cc6d6a-plugins-config-volume\") pod \"eb07ecde-e243-402e-965e-be6a87cc6d6a\" (UID: \"eb07ecde-e243-402e-965e-be6a87cc6d6a\") " Oct 31 22:20:03.822922 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.822902 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46nkm\" (UniqueName: \"kubernetes.io/projected/eb07ecde-e243-402e-965e-be6a87cc6d6a-kube-api-access-46nkm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:03.822987 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.822948 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb07ecde-e243-402e-965e-be6a87cc6d6a-plugins-config-volume" pod "eb07ecde-e243-402e-965e-be6a87cc6d6a" (UID: "eb07ecde-e243-402e-965e-be6a87cc6d6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:03.923470 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:03.923362 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb07ecde-e243-402e-965e-be6a87cc6d6a-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:04.516937 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.516891 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" event={"ID":"eb07ecde-e243-402e-965e-be6a87cc6d6a","Type":"ContainerDied","Data":"cd884cbfc6ed5e27f30e3517527531cdf4b44146d406fd786d9dce0c7f11b3ad"} Oct 31 22:20:04.517484 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.516977 5188 scope.go:122] "RemoveContainer" containerID="0c0ccb7736cb0d82a741e9d6c3d544725614156a894ba7bc2370d872978799ed" Oct 31 22:20:04.517484 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.516906 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd" Oct 31 22:20:04.562265 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.562224 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd"] Oct 31 22:20:04.573374 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.573331 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fb32d4-5b99d7f5ff-fp9xd"] Oct 31 22:20:04.682977 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.682945 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:04.731039 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.731002 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-openshift-ca\") pod \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " Oct 31 22:20:04.731191 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.731066 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-kube-api-access-mkfj6\" (UniqueName: \"kubernetes.io/projected/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-kube-api-access-mkfj6\") pod \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " Oct 31 22:20:04.731191 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.731158 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-xks-ca\") pod \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\" (UID: \"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392\") " Oct 31 22:20:04.731697 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.731660 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-openshift-ca" pod "ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392" (UID: "ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:04.736368 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.734042 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-xks-ca" pod "ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392" (UID: "ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:04.736368 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.734472 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-kube-api-access-mkfj6" pod "ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392" (UID: "ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392"). InnerVolumeSpecName "kube-api-access-mkfj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:04.832682 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.832598 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:04.832682 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.832629 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkfj6\" (UniqueName: \"kubernetes.io/projected/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-kube-api-access-mkfj6\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:04.832682 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:04.832641 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:05.472928 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:05.472893 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb07ecde-e243-402e-965e-be6a87cc6d6a" path="/var/lib/kubelet/pods/eb07ecde-e243-402e-965e-be6a87cc6d6a/volumes" Oct 31 22:20:05.524364 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:05.524310 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" Oct 31 22:20:05.524828 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:05.524307 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0b8f395-t6d9p" event={"ID":"ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392","Type":"ContainerDied","Data":"13b479be04be80887701d03a7c4bd09da05cf77bb851a6dd21eca4c936a3f4a1"} Oct 31 22:20:05.524828 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:05.524482 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13b479be04be80887701d03a7c4bd09da05cf77bb851a6dd21eca4c936a3f4a1" Oct 31 22:20:06.228356 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:06.228300 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" podUID="6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" containerName="payload-processing" containerID="cri-o://c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7" gracePeriod=25 Oct 31 22:20:06.264274 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:06.264223 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" podUID="b68f266a-8ca0-4e78-9e23-064ccc6e761c" containerName="payload-pre-processing" containerID="cri-o://13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5" gracePeriod=25 Oct 31 22:20:06.886620 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:06.886579 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" podUID="6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:20:06.887105 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:06.886630 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" podUID="b68f266a-8ca0-4e78-9e23-064ccc6e761c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:20:10.941347 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:10.940304 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m"] Oct 31 22:20:10.941347 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:10.940625 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" podUID="52ca6993-280b-4f66-b320-c2d9dca8a873" containerName="maas-api" containerID="cri-o://a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011" gracePeriod=30 Oct 31 22:20:11.220289 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.220252 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 22:20:11.297344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.296662 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-metrics-certs\") pod \"52ca6993-280b-4f66-b320-c2d9dca8a873\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " Oct 31 22:20:11.297344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.296781 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls\") pod \"52ca6993-280b-4f66-b320-c2d9dca8a873\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " Oct 31 22:20:11.297344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.296866 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ca6993-280b-4f66-b320-c2d9dca8a873-kube-api-access-mtlz9\" (UniqueName: \"kubernetes.io/projected/52ca6993-280b-4f66-b320-c2d9dca8a873-kube-api-access-mtlz9\") pod \"52ca6993-280b-4f66-b320-c2d9dca8a873\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " Oct 31 22:20:11.297344 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.296924 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ca6993-280b-4f66-b320-c2d9dca8a873-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca6993-280b-4f66-b320-c2d9dca8a873-openshift-service-ca\") pod \"52ca6993-280b-4f66-b320-c2d9dca8a873\" (UID: \"52ca6993-280b-4f66-b320-c2d9dca8a873\") " Oct 31 22:20:11.297753 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.297693 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ca6993-280b-4f66-b320-c2d9dca8a873-openshift-service-ca" pod "52ca6993-280b-4f66-b320-c2d9dca8a873" (UID: "52ca6993-280b-4f66-b320-c2d9dca8a873"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:11.302378 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.300898 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls" pod "52ca6993-280b-4f66-b320-c2d9dca8a873" (UID: "52ca6993-280b-4f66-b320-c2d9dca8a873"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:11.302378 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.301071 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-metrics-certs" pod "52ca6993-280b-4f66-b320-c2d9dca8a873" (UID: "52ca6993-280b-4f66-b320-c2d9dca8a873"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:11.302900 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.302859 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ca6993-280b-4f66-b320-c2d9dca8a873-kube-api-access-mtlz9" pod "52ca6993-280b-4f66-b320-c2d9dca8a873" (UID: "52ca6993-280b-4f66-b320-c2d9dca8a873"). InnerVolumeSpecName "kube-api-access-mtlz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:11.398025 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.397984 5188 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-metrics-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:11.398025 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.398018 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52ca6993-280b-4f66-b320-c2d9dca8a873-maas-api-tls\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:11.398025 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.398029 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtlz9\" (UniqueName: \"kubernetes.io/projected/52ca6993-280b-4f66-b320-c2d9dca8a873-kube-api-access-mtlz9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:11.398298 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.398039 5188 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52ca6993-280b-4f66-b320-c2d9dca8a873-openshift-service-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:11.565532 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.565492 5188 generic.go:356] "Generic (PLEG): container finished" podID="52ca6993-280b-4f66-b320-c2d9dca8a873" containerID="a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011" exitCode=0 Oct 31 22:20:11.565744 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.565552 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" Oct 31 22:20:11.565744 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.565572 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" event={"ID":"52ca6993-280b-4f66-b320-c2d9dca8a873","Type":"ContainerDied","Data":"a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011"} Oct 31 22:20:11.565744 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.565636 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m" event={"ID":"52ca6993-280b-4f66-b320-c2d9dca8a873","Type":"ContainerDied","Data":"815976be28c085bee082c668c113b3223408b458f55dfebab08870e88b67dda0"} Oct 31 22:20:11.565744 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.565664 5188 scope.go:122] "RemoveContainer" containerID="a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011" Oct 31 22:20:11.576705 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.576675 5188 scope.go:122] "RemoveContainer" containerID="a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011" Oct 31 22:20:11.577180 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:20:11.577149 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011\": container with ID starting with a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011 not found: ID does not exist" containerID="a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011" Oct 31 22:20:11.577280 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.577188 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011"} err="failed to get container status \"a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011\": rpc error: code = NotFound desc = could not find container \"a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011\": container with ID starting with a254ffdae194342b2a669c0641ec713bc4beba7bf41ab0114af6e4cac1307011 not found: ID does not exist" Oct 31 22:20:11.612684 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.612644 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m"] Oct 31 22:20:11.619394 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:11.619360 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-56a6d4-6bdbbf8d4d-dw47m"] Oct 31 22:20:13.474122 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:13.474074 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ca6993-280b-4f66-b320-c2d9dca8a873" path="/var/lib/kubelet/pods/52ca6993-280b-4f66-b320-c2d9dca8a873/volumes" Oct 31 22:20:16.886766 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:16.886723 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" podUID="6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:20:16.886766 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:16.886749 5188 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" podUID="b68f266a-8ca0-4e78-9e23-064ccc6e761c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:20:20.529306 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.529262 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd"] Oct 31 22:20:20.531034 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.531005 5188 cpu_manager.go:414] "containerMap: removing container" podUID="52ca6993-280b-4f66-b320-c2d9dca8a873" containerName="maas-api" Oct 31 22:20:20.531879 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.531858 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ca6993-280b-4f66-b320-c2d9dca8a873" containerName="maas-api" Oct 31 22:20:20.532025 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.532014 5188 cpu_manager.go:414] "containerMap: removing container" podUID="ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392" containerName="revoke-keys" Oct 31 22:20:20.532109 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.532099 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392" containerName="revoke-keys" Oct 31 22:20:20.532229 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.532220 5188 cpu_manager.go:414] "containerMap: removing container" podUID="eb07ecde-e243-402e-965e-be6a87cc6d6a" containerName="payload-pre-processing" Oct 31 22:20:20.532300 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.532290 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb07ecde-e243-402e-965e-be6a87cc6d6a" containerName="payload-pre-processing" Oct 31 22:20:20.532524 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.532511 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb07ecde-e243-402e-965e-be6a87cc6d6a" containerName="payload-pre-processing" Oct 31 22:20:20.532620 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.532611 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ca6993-280b-4f66-b320-c2d9dca8a873" containerName="maas-api" Oct 31 22:20:20.532695 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.532687 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392" containerName="revoke-keys" Oct 31 22:20:20.538014 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.537991 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.562384 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.562285 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 22:20:20.562583 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.562417 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 22:20:20.568182 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.568137 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd"] Oct 31 22:20:20.605661 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.605621 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bpzh\" (UniqueName: \"kubernetes.io/projected/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-kube-api-access-7bpzh\") pod \"maas-api-revoke-keys-e2e-worker-w3-106555-5msxd\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.605929 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.605694 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-106555-5msxd\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.605929 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.605843 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-106555-5msxd\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.618205 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.618164 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:20:20.618205 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.618191 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:20:20.707376 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.707318 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-106555-5msxd\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.707616 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.707408 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-106555-5msxd\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.707616 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.707493 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bpzh\" (UniqueName: \"kubernetes.io/projected/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-kube-api-access-7bpzh\") pod \"maas-api-revoke-keys-e2e-worker-w3-106555-5msxd\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.708103 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.708077 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-106555-5msxd\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.710427 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.710395 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-106555-5msxd\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.729272 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.729224 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bpzh\" (UniqueName: \"kubernetes.io/projected/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-kube-api-access-7bpzh\") pod \"maas-api-revoke-keys-e2e-worker-w3-106555-5msxd\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:20.859447 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:20.859277 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:21.162151 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:21.162116 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd"] Oct 31 22:20:21.164150 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:20:21.164115 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3e8a9c_cfbf_44c0_8e19_27e8e3817b0f.slice/crio-052d5e51c14717ced3a77d2cc535cc52c3998e55354292159d12f3fcb458801b WatchSource:0}: Error finding container 052d5e51c14717ced3a77d2cc535cc52c3998e55354292159d12f3fcb458801b: Status 404 returned error can't find the container with id 052d5e51c14717ced3a77d2cc535cc52c3998e55354292159d12f3fcb458801b Oct 31 22:20:21.630011 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:21.629967 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" event={"ID":"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f","Type":"ContainerStarted","Data":"b23c30b16f04ac48013e5f116ad2250f14be765054667b9c30a36db7dffd98b5"} Oct 31 22:20:21.630455 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:21.630039 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" event={"ID":"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f","Type":"ContainerStarted","Data":"052d5e51c14717ced3a77d2cc535cc52c3998e55354292159d12f3fcb458801b"} Oct 31 22:20:21.666747 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:21.666690 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" podStartSLOduration=1.666674889 podStartE2EDuration="1.666674889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:20:21.6661837 +0000 UTC m=+3868.359141093" watchObservedRunningTime="2026-10-31 22:20:21.666674889 +0000 UTC m=+3868.359632281" Oct 31 22:20:22.639566 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:22.639525 5188 generic.go:356] "Generic (PLEG): container finished" podID="bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f" containerID="b23c30b16f04ac48013e5f116ad2250f14be765054667b9c30a36db7dffd98b5" exitCode=0 Oct 31 22:20:22.645454 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:22.645406 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" event={"ID":"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f","Type":"ContainerDied","Data":"b23c30b16f04ac48013e5f116ad2250f14be765054667b9c30a36db7dffd98b5"} Oct 31 22:20:23.806029 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:23.805993 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:23.938881 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:23.938782 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-openshift-ca\") pod \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " Oct 31 22:20:23.939087 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:23.938897 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-kube-api-access-7bpzh\" (UniqueName: \"kubernetes.io/projected/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-kube-api-access-7bpzh\") pod \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " Oct 31 22:20:23.939087 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:23.938986 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-xks-ca\") pod \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\" (UID: \"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f\") " Oct 31 22:20:23.939240 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:23.939209 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-openshift-ca" pod "bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f" (UID: "bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:23.941064 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:23.941038 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-kube-api-access-7bpzh" pod "bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f" (UID: "bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f"). InnerVolumeSpecName "kube-api-access-7bpzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:23.941204 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:23.941180 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-xks-ca" pod "bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f" (UID: "bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:24.039973 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:24.039933 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bpzh\" (UniqueName: \"kubernetes.io/projected/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-kube-api-access-7bpzh\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:24.039973 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:24.039967 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:24.039973 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:24.039981 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:24.663288 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:24.663255 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" Oct 31 22:20:24.663504 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:24.663263 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd" event={"ID":"bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f","Type":"ContainerDied","Data":"052d5e51c14717ced3a77d2cc535cc52c3998e55354292159d12f3fcb458801b"} Oct 31 22:20:24.663504 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:24.663488 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="052d5e51c14717ced3a77d2cc535cc52c3998e55354292159d12f3fcb458801b" Oct 31 22:20:25.294050 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:25.294006 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd"] Oct 31 22:20:26.544382 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.544349 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 22:20:26.570118 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.570088 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 22:20:26.666703 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.666663 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68f266a-8ca0-4e78-9e23-064ccc6e761c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f266a-8ca0-4e78-9e23-064ccc6e761c-plugins-config-volume\") pod \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\" (UID: \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\") " Oct 31 22:20:26.666903 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.666719 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68f266a-8ca0-4e78-9e23-064ccc6e761c-kube-api-access-rblsx\" (UniqueName: \"kubernetes.io/projected/b68f266a-8ca0-4e78-9e23-064ccc6e761c-kube-api-access-rblsx\") pod \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\" (UID: \"b68f266a-8ca0-4e78-9e23-064ccc6e761c\") " Oct 31 22:20:26.666903 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.666766 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-kube-api-access-xf6l9\" (UniqueName: \"kubernetes.io/projected/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-kube-api-access-xf6l9\") pod \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\" (UID: \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\") " Oct 31 22:20:26.666903 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.666821 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-plugins-config-volume\") pod \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\" (UID: \"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9\") " Oct 31 22:20:26.667091 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.667057 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68f266a-8ca0-4e78-9e23-064ccc6e761c-plugins-config-volume" pod "b68f266a-8ca0-4e78-9e23-064ccc6e761c" (UID: "b68f266a-8ca0-4e78-9e23-064ccc6e761c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:26.667302 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.667278 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-plugins-config-volume" pod "6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" (UID: "6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:26.668944 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.668921 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68f266a-8ca0-4e78-9e23-064ccc6e761c-kube-api-access-rblsx" pod "b68f266a-8ca0-4e78-9e23-064ccc6e761c" (UID: "b68f266a-8ca0-4e78-9e23-064ccc6e761c"). InnerVolumeSpecName "kube-api-access-rblsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:26.669030 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.668944 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-kube-api-access-xf6l9" pod "6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" (UID: "6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9"). InnerVolumeSpecName "kube-api-access-xf6l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:26.678961 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.678926 5188 generic.go:356] "Generic (PLEG): container finished" podID="b68f266a-8ca0-4e78-9e23-064ccc6e761c" containerID="13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5" exitCode=0 Oct 31 22:20:26.679135 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.678976 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" Oct 31 22:20:26.679135 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.679012 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" event={"ID":"b68f266a-8ca0-4e78-9e23-064ccc6e761c","Type":"ContainerDied","Data":"13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5"} Oct 31 22:20:26.679135 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.679110 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz" event={"ID":"b68f266a-8ca0-4e78-9e23-064ccc6e761c","Type":"ContainerDied","Data":"9dc9738a4c9e075fbb920309c411391f33dcc95a9d844462bc7a1432d74cc328"} Oct 31 22:20:26.679296 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.679145 5188 scope.go:122] "RemoveContainer" containerID="13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5" Oct 31 22:20:26.680616 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.680588 5188 generic.go:356] "Generic (PLEG): container finished" podID="6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" containerID="c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7" exitCode=0 Oct 31 22:20:26.680701 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.680646 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" Oct 31 22:20:26.680750 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.680649 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" event={"ID":"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9","Type":"ContainerDied","Data":"c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7"} Oct 31 22:20:26.680785 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.680764 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt" event={"ID":"6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9","Type":"ContainerDied","Data":"dd0638baeeee0218bec88365670fe0de436af09b7545bdec98d80fce8f6bb56a"} Oct 31 22:20:26.691674 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.691642 5188 scope.go:122] "RemoveContainer" containerID="13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5" Oct 31 22:20:26.692103 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:20:26.692070 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5\": container with ID starting with 13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5 not found: ID does not exist" containerID="13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5" Oct 31 22:20:26.692229 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.692117 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5"} err="failed to get container status \"13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5\": rpc error: code = NotFound desc = could not find container \"13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5\": container with ID starting with 13407a09b63d53b06c86c0f75f6f23570e2858440db7ccac4696f20fa01a0dd5 not found: ID does not exist" Oct 31 22:20:26.692229 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.692144 5188 scope.go:122] "RemoveContainer" containerID="c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7" Oct 31 22:20:26.702256 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.702231 5188 scope.go:122] "RemoveContainer" containerID="c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7" Oct 31 22:20:26.702690 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:20:26.702665 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7\": container with ID starting with c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7 not found: ID does not exist" containerID="c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7" Oct 31 22:20:26.702770 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.702702 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7"} err="failed to get container status \"c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7\": rpc error: code = NotFound desc = could not find container \"c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7\": container with ID starting with c9f48febcb759730aa6c7265fff51715f9b74ccb456863c3b1b06d77011bc1e7 not found: ID does not exist" Oct 31 22:20:26.723292 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.723256 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz"] Oct 31 22:20:26.729948 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.729915 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-994336-55bfbff87f-whppz"] Oct 31 22:20:26.768487 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.768404 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf6l9\" (UniqueName: \"kubernetes.io/projected/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-kube-api-access-xf6l9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:26.768487 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.768431 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:26.768487 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.768443 5188 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b68f266a-8ca0-4e78-9e23-064ccc6e761c-plugins-config-volume\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:26.768487 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.768465 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rblsx\" (UniqueName: \"kubernetes.io/projected/b68f266a-8ca0-4e78-9e23-064ccc6e761c-kube-api-access-rblsx\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:26.770060 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.770030 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt"] Oct 31 22:20:26.774192 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:26.774157 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-994336-776fb75b4f-628xt"] Oct 31 22:20:27.473432 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:27.473394 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" path="/var/lib/kubelet/pods/6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9/volumes" Oct 31 22:20:27.473837 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:27.473820 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68f266a-8ca0-4e78-9e23-064ccc6e761c" path="/var/lib/kubelet/pods/b68f266a-8ca0-4e78-9e23-064ccc6e761c/volumes" Oct 31 22:20:32.538814 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:20:32.538765 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:20:33.415292 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.415247 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn"] Oct 31 22:20:33.415957 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.415935 5188 cpu_manager.go:414] "containerMap: removing container" podUID="bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f" containerName="revoke-keys" Oct 31 22:20:33.416015 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.415961 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f" containerName="revoke-keys" Oct 31 22:20:33.416015 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.415996 5188 cpu_manager.go:414] "containerMap: removing container" podUID="b68f266a-8ca0-4e78-9e23-064ccc6e761c" containerName="payload-pre-processing" Oct 31 22:20:33.416015 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.416003 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68f266a-8ca0-4e78-9e23-064ccc6e761c" containerName="payload-pre-processing" Oct 31 22:20:33.416125 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.416035 5188 cpu_manager.go:414] "containerMap: removing container" podUID="6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" containerName="payload-processing" Oct 31 22:20:33.416125 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.416042 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" containerName="payload-processing" Oct 31 22:20:33.416198 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.416146 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f" containerName="revoke-keys" Oct 31 22:20:33.416198 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.416164 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68f266a-8ca0-4e78-9e23-064ccc6e761c" containerName="payload-pre-processing" Oct 31 22:20:33.416198 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.416178 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc6ae49-47d0-4557-8f0e-7a8b32f3b3c9" containerName="payload-processing" Oct 31 22:20:33.420265 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.420236 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.435264 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.435230 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 22:20:33.445159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.445118 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:20:33.447532 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.447502 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:20:33.448104 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.448081 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn"] Oct 31 22:20:33.535207 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.535166 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 22:20:33.536612 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.536586 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f76dd9b-5g4sn\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.536690 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.536632 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26b9\" (UniqueName: \"kubernetes.io/projected/5480c96c-4173-4f6b-96ee-18337dea361f-kube-api-access-s26b9\") pod \"maas-api-revoke-sub-0f76dd9b-5g4sn\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.536792 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.536768 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f76dd9b-5g4sn\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.637649 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.637608 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26b9\" (UniqueName: \"kubernetes.io/projected/5480c96c-4173-4f6b-96ee-18337dea361f-kube-api-access-s26b9\") pod \"maas-api-revoke-sub-0f76dd9b-5g4sn\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.638159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.637677 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f76dd9b-5g4sn\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.638159 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.637757 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f76dd9b-5g4sn\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.638388 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.638364 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f76dd9b-5g4sn\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.640286 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.640258 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f76dd9b-5g4sn\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.659314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.659281 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26b9\" (UniqueName: \"kubernetes.io/projected/5480c96c-4173-4f6b-96ee-18337dea361f-kube-api-access-s26b9\") pod \"maas-api-revoke-sub-0f76dd9b-5g4sn\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:33.738168 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:33.738071 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:34.046967 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:34.046668 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn"] Oct 31 22:20:34.049027 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:20:34.048984 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5480c96c_4173_4f6b_96ee_18337dea361f.slice/crio-10d000c8a45e40cda7dbed763a5c6ef1f2193931c3218c9eaf5226ceff692a71 WatchSource:0}: Error finding container 10d000c8a45e40cda7dbed763a5c6ef1f2193931c3218c9eaf5226ceff692a71: Status 404 returned error can't find the container with id 10d000c8a45e40cda7dbed763a5c6ef1f2193931c3218c9eaf5226ceff692a71 Oct 31 22:20:34.736080 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:34.736033 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" event={"ID":"5480c96c-4173-4f6b-96ee-18337dea361f","Type":"ContainerStarted","Data":"28ac2453990d1bf00d3367d010b3edfddf0a49cef42d77b67cc82e816ad7e451"} Oct 31 22:20:34.736542 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:34.736104 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" event={"ID":"5480c96c-4173-4f6b-96ee-18337dea361f","Type":"ContainerStarted","Data":"10d000c8a45e40cda7dbed763a5c6ef1f2193931c3218c9eaf5226ceff692a71"} Oct 31 22:20:34.774405 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:34.774342 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" podStartSLOduration=1.774308566 podStartE2EDuration="1.774308566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:20:34.772879055 +0000 UTC m=+3881.465836448" watchObservedRunningTime="2026-10-31 22:20:34.774308566 +0000 UTC m=+3881.467265958" Oct 31 22:20:35.743096 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:35.743057 5188 generic.go:356] "Generic (PLEG): container finished" podID="5480c96c-4173-4f6b-96ee-18337dea361f" containerID="28ac2453990d1bf00d3367d010b3edfddf0a49cef42d77b67cc82e816ad7e451" exitCode=0 Oct 31 22:20:35.743544 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:35.743142 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" event={"ID":"5480c96c-4173-4f6b-96ee-18337dea361f","Type":"ContainerDied","Data":"28ac2453990d1bf00d3367d010b3edfddf0a49cef42d77b67cc82e816ad7e451"} Oct 31 22:20:36.888984 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:36.888959 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:37.071965 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.071868 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-xks-ca\") pod \"5480c96c-4173-4f6b-96ee-18337dea361f\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " Oct 31 22:20:37.071965 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.071914 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-openshift-ca\") pod \"5480c96c-4173-4f6b-96ee-18337dea361f\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " Oct 31 22:20:37.071965 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.071957 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5480c96c-4173-4f6b-96ee-18337dea361f-kube-api-access-s26b9\" (UniqueName: \"kubernetes.io/projected/5480c96c-4173-4f6b-96ee-18337dea361f-kube-api-access-s26b9\") pod \"5480c96c-4173-4f6b-96ee-18337dea361f\" (UID: \"5480c96c-4173-4f6b-96ee-18337dea361f\") " Oct 31 22:20:37.072437 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.072408 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-openshift-ca" pod "5480c96c-4173-4f6b-96ee-18337dea361f" (UID: "5480c96c-4173-4f6b-96ee-18337dea361f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:37.074207 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.074178 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-xks-ca" pod "5480c96c-4173-4f6b-96ee-18337dea361f" (UID: "5480c96c-4173-4f6b-96ee-18337dea361f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:37.074337 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.074230 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5480c96c-4173-4f6b-96ee-18337dea361f-kube-api-access-s26b9" pod "5480c96c-4173-4f6b-96ee-18337dea361f" (UID: "5480c96c-4173-4f6b-96ee-18337dea361f"). InnerVolumeSpecName "kube-api-access-s26b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:37.172987 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.172941 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-xks-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:37.172987 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.172978 5188 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5480c96c-4173-4f6b-96ee-18337dea361f-maas-api-openshift-ca\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:37.172987 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.172989 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s26b9\" (UniqueName: \"kubernetes.io/projected/5480c96c-4173-4f6b-96ee-18337dea361f-kube-api-access-s26b9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:37.755814 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.755715 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" event={"ID":"5480c96c-4173-4f6b-96ee-18337dea361f","Type":"ContainerDied","Data":"10d000c8a45e40cda7dbed763a5c6ef1f2193931c3218c9eaf5226ceff692a71"} Oct 31 22:20:37.755814 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.755772 5188 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10d000c8a45e40cda7dbed763a5c6ef1f2193931c3218c9eaf5226ceff692a71" Oct 31 22:20:37.755814 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:37.755732 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f76dd9b-5g4sn" Oct 31 22:20:40.316747 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.316698 5188 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" containerID="cri-o://e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8" gracePeriod=45 Oct 31 22:20:40.578812 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.578743 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 22:20:40.606645 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.606606 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-home\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-home\") pod \"9e0bae2f-c86a-487e-ac13-6b7890092979\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " Oct 31 22:20:40.606645 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.606648 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e0bae2f-c86a-487e-ac13-6b7890092979-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0bae2f-c86a-487e-ac13-6b7890092979-tls-certs\") pod \"9e0bae2f-c86a-487e-ac13-6b7890092979\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " Oct 31 22:20:40.606916 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.606677 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-model-cache\") pod \"9e0bae2f-c86a-487e-ac13-6b7890092979\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " Oct 31 22:20:40.606916 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.606745 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-tmp-dir\") pod \"9e0bae2f-c86a-487e-ac13-6b7890092979\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " Oct 31 22:20:40.606916 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.606778 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-dshm\") pod \"9e0bae2f-c86a-487e-ac13-6b7890092979\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " Oct 31 22:20:40.606916 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.606839 5188 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0bae2f-c86a-487e-ac13-6b7890092979-kube-api-access-4w9q9\" (UniqueName: \"kubernetes.io/projected/9e0bae2f-c86a-487e-ac13-6b7890092979-kube-api-access-4w9q9\") pod \"9e0bae2f-c86a-487e-ac13-6b7890092979\" (UID: \"9e0bae2f-c86a-487e-ac13-6b7890092979\") " Oct 31 22:20:40.607135 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.606965 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-home" pod "9e0bae2f-c86a-487e-ac13-6b7890092979" (UID: "9e0bae2f-c86a-487e-ac13-6b7890092979"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.607450 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.607202 5188 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-home\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.607574 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.607516 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-model-cache" pod "9e0bae2f-c86a-487e-ac13-6b7890092979" (UID: "9e0bae2f-c86a-487e-ac13-6b7890092979"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.607725 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.607693 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-tmp-dir" pod "9e0bae2f-c86a-487e-ac13-6b7890092979" (UID: "9e0bae2f-c86a-487e-ac13-6b7890092979"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.609930 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.609890 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e0bae2f-c86a-487e-ac13-6b7890092979-tls-certs" pod "9e0bae2f-c86a-487e-ac13-6b7890092979" (UID: "9e0bae2f-c86a-487e-ac13-6b7890092979"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:40.610460 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.610431 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-dshm" pod "9e0bae2f-c86a-487e-ac13-6b7890092979" (UID: "9e0bae2f-c86a-487e-ac13-6b7890092979"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.610693 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.610665 5188 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0bae2f-c86a-487e-ac13-6b7890092979-kube-api-access-4w9q9" pod "9e0bae2f-c86a-487e-ac13-6b7890092979" (UID: "9e0bae2f-c86a-487e-ac13-6b7890092979"). InnerVolumeSpecName "kube-api-access-4w9q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:40.708620 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.708577 5188 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-tmp-dir\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.708620 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.708621 5188 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-dshm\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.708859 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.708636 5188 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w9q9\" (UniqueName: \"kubernetes.io/projected/9e0bae2f-c86a-487e-ac13-6b7890092979-kube-api-access-4w9q9\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.708859 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.708708 5188 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e0bae2f-c86a-487e-ac13-6b7890092979-tls-certs\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.708859 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.708718 5188 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e0bae2f-c86a-487e-ac13-6b7890092979-model-cache\") on node \"ip-10-0-131-21.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.776608 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.776566 5188 generic.go:356] "Generic (PLEG): container finished" podID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerID="e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8" exitCode=0 Oct 31 22:20:40.776786 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.776613 5188 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" Oct 31 22:20:40.776786 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.776651 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" event={"ID":"9e0bae2f-c86a-487e-ac13-6b7890092979","Type":"ContainerDied","Data":"e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8"} Oct 31 22:20:40.776786 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.776706 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd" event={"ID":"9e0bae2f-c86a-487e-ac13-6b7890092979","Type":"ContainerDied","Data":"819cc292d0e660b70f724a5c574c59a8afbac3eee537c0c9d85eb5521b410592"} Oct 31 22:20:40.776786 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.776729 5188 scope.go:122] "RemoveContainer" containerID="e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8" Oct 31 22:20:40.788998 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.788975 5188 scope.go:122] "RemoveContainer" containerID="e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8" Oct 31 22:20:40.789367 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:20:40.789314 5188 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8\": container with ID starting with e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8 not found: ID does not exist" containerID="e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8" Oct 31 22:20:40.789468 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.789380 5188 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8"} err="failed to get container status \"e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8\": rpc error: code = NotFound desc = could not find container \"e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8\": container with ID starting with e47bb936260839e0501f8bbf627990e60668e5d01c038412c3ac6bc3e9b2d4e8 not found: ID does not exist" Oct 31 22:20:40.824966 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.824921 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd"] Oct 31 22:20:40.830086 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:40.829998 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-106555/premium-w3-106555-kserve-7ff9cb8b77-npmzd"] Oct 31 22:20:41.472260 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:41.472219 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" path="/var/lib/kubelet/pods/9e0bae2f-c86a-487e-ac13-6b7890092979/volumes" Oct 31 22:20:49.152571 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:49.152532 5188 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd"] Oct 31 22:20:49.162180 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:49.162142 5188 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-106555-5msxd"] Oct 31 22:20:49.472084 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:49.471996 5188 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f" path="/var/lib/kubelet/pods/bc3e8a9c-cfbf-44c0-8e19-27e8e3817b0f/volumes" Oct 31 22:20:54.257143 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:54.257115 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-k4qxr_e49a2d50-dc82-4983-9425-053dfe903ddd/cluster-samples-operator/0.log" Oct 31 22:20:54.260409 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:54.260384 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:20:54.266314 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:54.266286 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6ddb6c8b94-k4qxr_e49a2d50-dc82-4983-9425-053dfe903ddd/cluster-samples-operator/0.log" Oct 31 22:20:54.269290 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:54.269265 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:20:57.776023 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:57.775929 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-l7tsw_71e739c4-f697-4b84-9b3f-1ee74b73a01b/manager/0.log" Oct 31 22:20:58.557014 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:58.556978 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cxhcb_e8d62927-0be1-4ab5-8961-f28340b349c3/manager/0.log" Oct 31 22:20:59.933789 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:20:59.933757 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zpt8v_c6142fdd-301b-49dc-aa47-689dcfe7c061/manager/0.log" Oct 31 22:21:00.143725 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:00.143674 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j48f9_ae34d5ae-7e14-4a0b-b89b-d22c254f7618/manager/0.log" Oct 31 22:21:00.874283 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:00.874247 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/1.log" Oct 31 22:21:01.050090 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:01.050059 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-25bhl_ed5c581e-3188-4ecf-acb4-6c0fc04dc146/server/0.log" Oct 31 22:21:01.237447 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:01.237369 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5wlcw_4a78bc84-6872-4292-b6f5-60e3737194a7/manager/0.log" Oct 31 22:21:01.446479 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:01.446444 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hbkp4_335b5cbb-fd40-44fd-9d2f-ec66c41cae34/proxy/0.log" Oct 31 22:21:01.849012 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:01.848978 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/0.log" Oct 31 22:21:02.538387 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:21:02.538318 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:21:12.988879 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:12.988782 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zpt8v_c6142fdd-301b-49dc-aa47-689dcfe7c061/manager/0.log" Oct 31 22:21:13.176283 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:13.176247 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j48f9_ae34d5ae-7e14-4a0b-b89b-d22c254f7618/manager/0.log" Oct 31 22:21:13.918915 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:13.918879 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/1.log" Oct 31 22:21:14.109907 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:14.109876 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-25bhl_ed5c581e-3188-4ecf-acb4-6c0fc04dc146/server/0.log" Oct 31 22:21:14.310242 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:14.310157 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5wlcw_4a78bc84-6872-4292-b6f5-60e3737194a7/manager/0.log" Oct 31 22:21:14.546852 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:14.546814 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hbkp4_335b5cbb-fd40-44fd-9d2f-ec66c41cae34/proxy/0.log" Oct 31 22:21:14.953063 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:14.953026 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/0.log" Oct 31 22:21:15.917989 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:15.917948 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0f76dd9b-5g4sn_5480c96c-4173-4f6b-96ee-18337dea361f/revoke-keys/0.log" Oct 31 22:21:17.823025 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:17.822996 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d0b8f395-t6d9p_ff750fd3-bf7c-45a5-bbb2-04dd0d8f0392/revoke-keys/0.log" Oct 31 22:21:19.726915 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:19.726878 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-l7tsw_71e739c4-f697-4b84-9b3f-1ee74b73a01b/manager/0.log" Oct 31 22:21:20.442068 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:20.442023 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cxhcb_e8d62927-0be1-4ab5-8961-f28340b349c3/manager/0.log" Oct 31 22:21:22.208521 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:22.208485 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5587c8dd5c-htpqs_dc9113e0-2ee8-44de-af57-00602a936fe6/kube-auth-proxy/0.log" Oct 31 22:21:22.415347 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:22.415296 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-btqnt_cc92a890-3858-49b3-af22-c1f80c348119/istio-proxy/0.log" Oct 31 22:21:23.202023 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:23.201979 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-f9sls_fe61268d-fbb8-4676-80cf-0309e133856e/payload-pre-processing/0.log" Oct 31 22:21:23.576661 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:23.576576 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d4c65fcc-47wdn_934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/router/0.log" Oct 31 22:21:24.321913 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:24.321873 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-2trls_8fae092c-44ce-4bc7-9c06-5f946eee3d2a/main/0.log" Oct 31 22:21:25.089342 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:25.089306 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8j9gvf_7c844161-8129-4828-9e4c-2cc46c64adca/main/0.log" Oct 31 22:21:25.279646 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:25.279598 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-4gck9_a4686a76-fe03-456a-a669-6f8a1a6bbcf8/main/0.log" Oct 31 22:21:25.457958 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:25.457923 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lq8t9_a155f827-18b2-4296-bd24-4584460f57d4/main/0.log" Oct 31 22:21:32.538712 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:21:32.538667 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:21:35.849724 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.849687 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2272/must-gather-rtxqb"] Oct 31 22:21:35.850149 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.850106 5188 cpu_manager.go:414] "containerMap: removing container" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" Oct 31 22:21:35.850149 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.850117 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" Oct 31 22:21:35.850241 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.850153 5188 cpu_manager.go:414] "containerMap: removing container" podUID="5480c96c-4173-4f6b-96ee-18337dea361f" containerName="revoke-keys" Oct 31 22:21:35.850241 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.850159 5188 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5480c96c-4173-4f6b-96ee-18337dea361f" containerName="revoke-keys" Oct 31 22:21:35.850241 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.850220 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0bae2f-c86a-487e-ac13-6b7890092979" containerName="main" Oct 31 22:21:35.850241 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.850234 5188 memory_manager.go:365] "RemoveStaleState removing state" podUID="5480c96c-4173-4f6b-96ee-18337dea361f" containerName="revoke-keys" Oct 31 22:21:35.853595 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.853575 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2272/must-gather-rtxqb" Oct 31 22:21:35.872224 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.872186 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2272/must-gather-rtxqb"] Oct 31 22:21:35.875572 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.875545 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2272\"/\"kube-root-ca.crt\"" Oct 31 22:21:35.875572 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.875555 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2272\"/\"openshift-service-ca.crt\"" Oct 31 22:21:35.875803 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.875615 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f2272\"/\"default-dockercfg-mczmx\"" Oct 31 22:21:35.928845 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.928815 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b50accd3-90e2-411b-9a83-9e2397c14add-must-gather-output\") pod \"must-gather-rtxqb\" (UID: \"b50accd3-90e2-411b-9a83-9e2397c14add\") " pod="openshift-must-gather-f2272/must-gather-rtxqb" Oct 31 22:21:35.929054 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:35.928951 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2c9r\" (UniqueName: \"kubernetes.io/projected/b50accd3-90e2-411b-9a83-9e2397c14add-kube-api-access-t2c9r\") pod \"must-gather-rtxqb\" (UID: \"b50accd3-90e2-411b-9a83-9e2397c14add\") " pod="openshift-must-gather-f2272/must-gather-rtxqb" Oct 31 22:21:36.030057 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:36.030004 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b50accd3-90e2-411b-9a83-9e2397c14add-must-gather-output\") pod \"must-gather-rtxqb\" (UID: \"b50accd3-90e2-411b-9a83-9e2397c14add\") " pod="openshift-must-gather-f2272/must-gather-rtxqb" Oct 31 22:21:36.030279 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:36.030096 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2c9r\" (UniqueName: \"kubernetes.io/projected/b50accd3-90e2-411b-9a83-9e2397c14add-kube-api-access-t2c9r\") pod \"must-gather-rtxqb\" (UID: \"b50accd3-90e2-411b-9a83-9e2397c14add\") " pod="openshift-must-gather-f2272/must-gather-rtxqb" Oct 31 22:21:36.030421 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:36.030398 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b50accd3-90e2-411b-9a83-9e2397c14add-must-gather-output\") pod \"must-gather-rtxqb\" (UID: \"b50accd3-90e2-411b-9a83-9e2397c14add\") " pod="openshift-must-gather-f2272/must-gather-rtxqb" Oct 31 22:21:36.051433 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:36.051398 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2c9r\" (UniqueName: \"kubernetes.io/projected/b50accd3-90e2-411b-9a83-9e2397c14add-kube-api-access-t2c9r\") pod \"must-gather-rtxqb\" (UID: \"b50accd3-90e2-411b-9a83-9e2397c14add\") " pod="openshift-must-gather-f2272/must-gather-rtxqb" Oct 31 22:21:36.169066 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:36.169029 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2272/must-gather-rtxqb" Oct 31 22:21:36.473335 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:36.473292 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2272/must-gather-rtxqb"] Oct 31 22:21:36.475524 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:21:36.475492 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb50accd3_90e2_411b_9a83_9e2397c14add.slice/crio-8170714320daebb5de8a43e95c623be972bc8bc54daed58e8d38b055d8a1e96b WatchSource:0}: Error finding container 8170714320daebb5de8a43e95c623be972bc8bc54daed58e8d38b055d8a1e96b: Status 404 returned error can't find the container with id 8170714320daebb5de8a43e95c623be972bc8bc54daed58e8d38b055d8a1e96b Oct 31 22:21:36.477493 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:36.477475 5188 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 22:21:37.119178 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:37.119139 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2272/must-gather-rtxqb" event={"ID":"b50accd3-90e2-411b-9a83-9e2397c14add","Type":"ContainerStarted","Data":"8170714320daebb5de8a43e95c623be972bc8bc54daed58e8d38b055d8a1e96b"} Oct 31 22:21:40.820711 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:40.820679 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-whzgx_72751047-bc3c-4f2d-9655-e23d5211d346/global-pull-secret-syncer/0.log" Oct 31 22:21:41.004493 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:41.004459 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7fh65_4bf7148c-69be-45a9-bd56-7457a3929df8/connection-checker/0.log" Oct 31 22:21:41.135398 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:41.135358 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cps44_7ed572b5-9b0c-4924-af2f-87703ed1b5e6/connection-checker/0.log" Oct 31 22:21:41.267511 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:41.267476 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-w96gb_9f8d2547-a263-47ff-aaa9-780ea67176a5/connection-checker/0.log" Oct 31 22:21:41.564739 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:41.564668 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9kbsn_ac6429fb-48a7-4691-9079-4ed168f1da32/konnectivity-agent/0.log" Oct 31 22:21:41.859135 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:41.859105 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-21.ec2.internal_d208acf97ac3e1709a838235c7640507/haproxy/0.log" Oct 31 22:21:42.159255 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:42.159208 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2272/must-gather-rtxqb" event={"ID":"b50accd3-90e2-411b-9a83-9e2397c14add","Type":"ContainerStarted","Data":"84e1d9eaf02813daca7ba46902e65cb16b6fd3a1d83e7921a3830bb9e048cbe1"} Oct 31 22:21:42.159486 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:42.159271 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2272/must-gather-rtxqb" event={"ID":"b50accd3-90e2-411b-9a83-9e2397c14add","Type":"ContainerStarted","Data":"28cfa63607c9e28ced65a320b60cd32168321d0c05e995f4febc9726c40eb537"} Oct 31 22:21:42.196840 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:42.196775 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-f2272/must-gather-rtxqb" podStartSLOduration=2.023858159 podStartE2EDuration="7.196759554s" totalImagesPullingTime="5.172901395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:21:42.19484664 +0000 UTC m=+3948.887804034" watchObservedRunningTime="2026-10-31 22:21:42.196759554 +0000 UTC m=+3948.889716946" Oct 31 22:21:43.693671 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:43.693635 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zpt8v_c6142fdd-301b-49dc-aa47-689dcfe7c061/manager/0.log" Oct 31 22:21:43.800307 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:43.800262 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j48f9_ae34d5ae-7e14-4a0b-b89b-d22c254f7618/manager/0.log" Oct 31 22:21:45.397571 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:45.397525 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/0.log" Oct 31 22:21:45.782568 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:45.782479 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/1.log" Oct 31 22:21:45.818302 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:45.818265 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-25bhl_ed5c581e-3188-4ecf-acb4-6c0fc04dc146/server/0.log" Oct 31 22:21:47.064785 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:47.064749 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5wlcw_4a78bc84-6872-4292-b6f5-60e3737194a7/manager/0.log" Oct 31 22:21:47.109609 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:47.109574 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hbkp4_335b5cbb-fd40-44fd-9d2f-ec66c41cae34/proxy/0.log" Oct 31 22:21:47.999267 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:47.999235 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-l7tsw_71e739c4-f697-4b84-9b3f-1ee74b73a01b/manager/0.log" Oct 31 22:21:48.463143 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:48.463105 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cxhcb_e8d62927-0be1-4ab5-8961-f28340b349c3/manager/0.log" Oct 31 22:21:48.509882 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:48.509852 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zpt8v_c6142fdd-301b-49dc-aa47-689dcfe7c061/manager/0.log" Oct 31 22:21:48.620857 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:48.620819 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j48f9_ae34d5ae-7e14-4a0b-b89b-d22c254f7618/manager/0.log" Oct 31 22:21:50.173744 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.173711 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/0.log" Oct 31 22:21:50.546880 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.546802 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/1.log" Oct 31 22:21:50.599673 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.599640 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-25bhl_ed5c581e-3188-4ecf-acb4-6c0fc04dc146/server/0.log" Oct 31 22:21:50.756393 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.756351 5188 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4"] Oct 31 22:21:50.764852 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.764816 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.780530 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.780497 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bgz52\"/\"kube-root-ca.crt\"" Oct 31 22:21:50.788915 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.788881 5188 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bgz52\"/\"openshift-service-ca.crt\"" Oct 31 22:21:50.789077 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.788966 5188 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bgz52\"/\"default-dockercfg-8gbfk\"" Oct 31 22:21:50.802080 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.801994 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4"] Oct 31 22:21:50.880052 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.880013 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-proc\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.880259 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.880081 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-lib-modules\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.880259 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.880134 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-podres\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.880259 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.880238 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-sys\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.880417 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.880279 5188 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sslpq\" (UniqueName: \"kubernetes.io/projected/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-kube-api-access-sslpq\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.990830 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.990782 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-lib-modules\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.991064 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.990946 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-lib-modules\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.991064 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.991013 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-podres\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.991196 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.991064 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-sys\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.991196 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.991102 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sslpq\" (UniqueName: \"kubernetes.io/projected/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-kube-api-access-sslpq\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.991196 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.991181 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-podres\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.991439 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.991234 5188 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-proc\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.991439 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.991277 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-sys\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:50.991570 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:50.991470 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-proc\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:51.013281 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:51.013238 5188 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sslpq\" (UniqueName: \"kubernetes.io/projected/bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0-kube-api-access-sslpq\") pod \"perf-node-gather-daemonset-w6qh4\" (UID: \"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:51.080111 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:51.080019 5188 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:51.394627 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:51.394552 5188 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4"] Oct 31 22:21:51.397902 ip-10-0-131-21 kubenswrapper[5188]: W1031 22:21:51.397868 5188 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbcdb1c11_2ac3_42c6_bfb8_78a98f6e90c0.slice/crio-7d0948f6021e0e8a6715649bacb91ddf5d56832738503361b4dbc062ed51c2fb WatchSource:0}: Error finding container 7d0948f6021e0e8a6715649bacb91ddf5d56832738503361b4dbc062ed51c2fb: Status 404 returned error can't find the container with id 7d0948f6021e0e8a6715649bacb91ddf5d56832738503361b4dbc062ed51c2fb Oct 31 22:21:51.786840 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:51.786746 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-8r9vm_c01f3957-4a49-41ba-812b-ea2e74d3a7a8/cluster-monitoring-operator/0.log" Oct 31 22:21:51.813625 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:51.813588 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5wlcw_4a78bc84-6872-4292-b6f5-60e3737194a7/manager/0.log" Oct 31 22:21:51.897056 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:51.897023 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hbkp4_335b5cbb-fd40-44fd-9d2f-ec66c41cae34/proxy/0.log" Oct 31 22:21:52.225220 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:52.225178 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" event={"ID":"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0","Type":"ContainerStarted","Data":"a5fa4660bc686d234c9e009b08fc019d59051be91740fb0836332ca2fe558abf"} Oct 31 22:21:52.225488 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:52.225234 5188 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" event={"ID":"bcdb1c11-2ac3-42c6-bfb8-78a98f6e90c0","Type":"ContainerStarted","Data":"7d0948f6021e0e8a6715649bacb91ddf5d56832738503361b4dbc062ed51c2fb"} Oct 31 22:21:52.225488 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:52.225365 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:52.262768 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:52.262707 5188 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" podStartSLOduration=2.262692779 podStartE2EDuration="2.262692779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:21:52.260390093 +0000 UTC m=+3958.953347485" watchObservedRunningTime="2026-10-31 22:21:52.262692779 +0000 UTC m=+3958.955650205" Oct 31 22:21:52.548244 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:52.548158 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hdr5_4d774b4f-526b-4b95-9620-7bfa56cb12a6/node-exporter/0.log" Oct 31 22:21:52.675070 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:52.675040 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hdr5_4d774b4f-526b-4b95-9620-7bfa56cb12a6/kube-rbac-proxy/0.log" Oct 31 22:21:52.796088 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:52.796059 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7hdr5_4d774b4f-526b-4b95-9620-7bfa56cb12a6/init-textfile/0.log" Oct 31 22:21:53.267172 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:53.267140 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-zpt8v_c6142fdd-301b-49dc-aa47-689dcfe7c061/manager/0.log" Oct 31 22:21:53.371736 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:53.371698 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-j48f9_ae34d5ae-7e14-4a0b-b89b-d22c254f7618/manager/0.log" Oct 31 22:21:54.495502 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:54.495466 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/0.log" Oct 31 22:21:54.855709 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:54.855551 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5b9c46cb8-xlhtv_9d1ac43e-7514-4d51-947a-f3bf325f2a94/manager/1.log" Oct 31 22:21:55.073277 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:55.073214 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-25bhl_ed5c581e-3188-4ecf-acb4-6c0fc04dc146/server/0.log" Oct 31 22:21:56.223837 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:56.223799 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-5wlcw_4a78bc84-6872-4292-b6f5-60e3737194a7/manager/0.log" Oct 31 22:21:56.354216 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:56.354166 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hbkp4_335b5cbb-fd40-44fd-9d2f-ec66c41cae34/proxy/0.log" Oct 31 22:21:57.872772 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:57.872729 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-sh7vx_0bd131db-e6d0-46c6-9a4b-85d3b541a9ec/networking-console-plugin/0.log" Oct 31 22:21:58.243770 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:58.243685 5188 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-w6qh4" Oct 31 22:21:58.945745 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:58.945707 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/3.log" Oct 31 22:21:58.950049 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:21:58.950017 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cg8tz_1b23775a-33a5-47b0-ac74-fb3862912542/console-operator/4.log" Oct 31 22:22:01.374084 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:01.374046 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-swfql_f7d03911-7706-47b8-928e-6ffea21fdb26/volume-data-source-validator/0.log" Oct 31 22:22:02.538438 ip-10-0-131-21 kubenswrapper[5188]: E1031 22:22:02.538398 5188 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dhxqp" containerName="extractor" Oct 31 22:22:03.148597 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:03.148562 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5587c8dd5c-htpqs_dc9113e0-2ee8-44de-af57-00602a936fe6/kube-auth-proxy/0.log" Oct 31 22:22:03.217774 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:03.217744 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-btqnt_cc92a890-3858-49b3-af22-c1f80c348119/istio-proxy/0.log" Oct 31 22:22:03.477595 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:03.477499 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-f9sls_fe61268d-fbb8-4676-80cf-0309e133856e/payload-pre-processing/0.log" Oct 31 22:22:03.610675 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:03.610636 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-68d4c65fcc-47wdn_934a6c09-d6c4-4ae2-afbf-3bc19cc948c8/router/0.log" Oct 31 22:22:04.242780 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:04.242735 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wj49j_b6f98741-1d48-4124-9acb-8c0bf31dbe60/dns/0.log" Oct 31 22:22:04.374121 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:04.374080 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wj49j_b6f98741-1d48-4124-9acb-8c0bf31dbe60/kube-rbac-proxy/0.log" Oct 31 22:22:04.622553 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:04.622520 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vx79w_6805be97-0a2e-490c-a50c-6700f15c4abc/dns-node-resolver/0.log" Oct 31 22:22:06.148450 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:06.148403 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-42vxn_672a2ce2-d45d-4f0c-a75a-b018105640aa/node-ca/0.log" Oct 31 22:22:07.063841 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:07.063798 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-l7tsw_71e739c4-f697-4b84-9b3f-1ee74b73a01b/manager/0.log" Oct 31 22:22:07.328497 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:07.328401 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-cxhcb_e8d62927-0be1-4ab5-8961-f28340b349c3/manager/0.log" Oct 31 22:22:09.530792 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:09.530754 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5587c8dd5c-htpqs_dc9113e0-2ee8-44de-af57-00602a936fe6/kube-auth-proxy/0.log" Oct 31 22:22:09.602296 ip-10-0-131-21 kubenswrapper[5188]: I1031 22:22:09.602258 5188 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-btqnt_cc92a890-3858-49b3-af22-c1f80c348119/istio-proxy/0.log"